Note: This is test shard 2 of 8.
[==========] Running 8 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from RebalanceIgnoredTserversTest
[ RUN ] RebalanceIgnoredTserversTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 08:10:41.093164 21842 test_util.cc:268] Using random seed: 96529922
I20240501 08:10:41.104561 21842 ts_itest-base.cc:112] Starting cluster with:
I20240501 08:10:41.104648 21842 ts_itest-base.cc:113] --------------
I20240501 08:10:41.104693 21842 ts_itest-base.cc:114] 5 tablet servers
I20240501 08:10:41.104756 21842 ts_itest-base.cc:115] 3 replicas per TS
I20240501 08:10:41.104822 21842 ts_itest-base.cc:116] --------------
2024-05-01T08:10:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T08:10:41Z Disabled control of system clock
I20240501 08:10:41.130718 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.84.190:46653
--webserver_interface=127.21.84.190
--webserver_port=0
--builtin_ntp_servers=127.21.84.148:36877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.84.190:46653
--enable_range_replica_placement=false with env {}
W20240501 08:10:41.450870 21854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:41.451201 21854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:41.451324 21854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:41.458828 21854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 08:10:41.458963 21854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:41.459039 21854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 08:10:41.459123 21854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 08:10:41.468250 21854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:36877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.21.84.190:46653
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.84.190:46653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.21.84.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:41.469653 21854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:41.471366 21854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:41.482265 21859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:41.483021 21862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:41.483650 21860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:41.484274 21854 server_base.cc:1034] running on GCE node
I20240501 08:10:41.485070 21854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:41.486578 21854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:41.487820 21854 hybrid_clock.cc:648] HybridClock initialized: now 1714551041487757 us; error 49 us; skew 500 ppm
I20240501 08:10:41.488165 21854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:41.494794 21854 webserver.cc:460] Webserver started at http://127.21.84.190:42225/ using document root <none> and password file <none>
I20240501 08:10:41.495608 21854 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:41.495754 21854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:41.496065 21854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:41.498826 21854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "4bda059414f84458af572d759dd7a451"
format_stamp: "Formatted at 2024-05-01 08:10:41 on dist-test-slave-9cgk"
I20240501 08:10:41.499540 21854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "4bda059414f84458af572d759dd7a451"
format_stamp: "Formatted at 2024-05-01 08:10:41 on dist-test-slave-9cgk"
I20240501 08:10:41.504894 21854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20240501 08:10:41.508415 21868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:41.509763 21854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 08:10:41.509994 21854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4bda059414f84458af572d759dd7a451"
format_stamp: "Formatted at 2024-05-01 08:10:41 on dist-test-slave-9cgk"
I20240501 08:10:41.510179 21854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:41.528259 21854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:41.529142 21854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:41.529431 21854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:41.547150 21854 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.190:46653
I20240501 08:10:41.547170 21919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.190:46653 every 8 connection(s)
I20240501 08:10:41.548586 21854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 08:10:41.552372 21920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:41.559155 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 21854
I20240501 08:10:41.559448 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 08:10:41.563031 21920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bda059414f84458af572d759dd7a451: Bootstrap starting.
I20240501 08:10:41.567099 21920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bda059414f84458af572d759dd7a451: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:41.568256 21920 log.cc:824] T 00000000000000000000000000000000 P 4bda059414f84458af572d759dd7a451: Log is configured to *not* fsync() on all Append() calls
I20240501 08:10:41.570967 21920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bda059414f84458af572d759dd7a451: No bootstrap required, opened a new log
I20240501 08:10:41.578066 21920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bda059414f84458af572d759dd7a451 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bda059414f84458af572d759dd7a451" member_type: VOTER last_known_addr { host: "127.21.84.190" port: 46653 } }
I20240501 08:10:41.578579 21920 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bda059414f84458af572d759dd7a451 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:41.578707 21920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bda059414f84458af572d759dd7a451 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bda059414f84458af572d759dd7a451, State: Initialized, Role: FOLLOWER
I20240501 08:10:41.579357 21920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bda059414f84458af572d759dd7a451 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bda059414f84458af572d759dd7a451" member_type: VOTER last_known_addr { host: "127.21.84.190" port: 46653 } }
I20240501 08:10:41.579555 21920 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bda059414f84458af572d759dd7a451 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:10:41.579689 21920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bda059414f84458af572d759dd7a451 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:10:41.579845 21920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bda059414f84458af572d759dd7a451 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:41.582885 21920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bda059414f84458af572d759dd7a451 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bda059414f84458af572d759dd7a451" member_type: VOTER last_known_addr { host: "127.21.84.190" port: 46653 } }
I20240501 08:10:41.583374 21920 leader_election.cc:302] T 00000000000000000000000000000000 P 4bda059414f84458af572d759dd7a451 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bda059414f84458af572d759dd7a451; no voters:
I20240501 08:10:41.584151 21920 leader_election.cc:288] T 00000000000000000000000000000000 P 4bda059414f84458af572d759dd7a451 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:10:41.584340 21925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bda059414f84458af572d759dd7a451 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:41.584997 21925 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bda059414f84458af572d759dd7a451 [term 1 LEADER]: Becoming Leader. State: Replica: 4bda059414f84458af572d759dd7a451, State: Running, Role: LEADER
I20240501 08:10:41.585644 21925 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bda059414f84458af572d759dd7a451 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bda059414f84458af572d759dd7a451" member_type: VOTER last_known_addr { host: "127.21.84.190" port: 46653 } }
I20240501 08:10:41.586810 21920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bda059414f84458af572d759dd7a451 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:10:41.588760 21927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bda059414f84458af572d759dd7a451 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bda059414f84458af572d759dd7a451. Latest consensus state: current_term: 1 leader_uuid: "4bda059414f84458af572d759dd7a451" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bda059414f84458af572d759dd7a451" member_type: VOTER last_known_addr { host: "127.21.84.190" port: 46653 } } }
I20240501 08:10:41.589958 21927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bda059414f84458af572d759dd7a451 [sys.catalog]: This master's current role is: LEADER
I20240501 08:10:41.589660 21926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bda059414f84458af572d759dd7a451 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bda059414f84458af572d759dd7a451" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bda059414f84458af572d759dd7a451" member_type: VOTER last_known_addr { host: "127.21.84.190" port: 46653 } } }
I20240501 08:10:41.590358 21926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bda059414f84458af572d759dd7a451 [sys.catalog]: This master's current role is: LEADER
I20240501 08:10:41.592147 21931 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:10:41.597637 21931 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:10:41.606885 21931 catalog_manager.cc:1347] Generated new cluster ID: 81a80eb1a228423eb0fa7a58371b1a48
I20240501 08:10:41.607017 21931 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:10:41.636286 21931 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:10:41.638090 21931 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:10:41.653878 21931 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 4bda059414f84458af572d759dd7a451: Generated new TSK 0
I20240501 08:10:41.654987 21931 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:10:41.660091 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.129:0
--local_ip_for_outbound_sockets=127.21.84.129
--webserver_interface=127.21.84.129
--webserver_port=0
--tserver_master_addrs=127.21.84.190:46653
--builtin_ntp_servers=127.21.84.148:36877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240501 08:10:41.983211 21944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:41.983534 21944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:41.983700 21944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:41.991420 21944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:41.991675 21944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.129
I20240501 08:10:42.000968 21944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:36877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.21.84.129
--webserver_port=0
--tserver_master_addrs=127.21.84.190:46653
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.129
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:42.002430 21944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:42.004184 21944 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:42.016340 21950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:42.017035 21949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:42.017715 21952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:42.018445 21944 server_base.cc:1034] running on GCE node
I20240501 08:10:42.019073 21944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:42.019980 21944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:42.021195 21944 hybrid_clock.cc:648] HybridClock initialized: now 1714551042021119 us; error 53 us; skew 500 ppm
I20240501 08:10:42.021490 21944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:42.023996 21944 webserver.cc:460] Webserver started at http://127.21.84.129:44841/ using document root <none> and password file <none>
I20240501 08:10:42.024732 21944 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:42.024855 21944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:42.025136 21944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:42.028118 21944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "6877815dd0594c039522791c8671008e"
format_stamp: "Formatted at 2024-05-01 08:10:42 on dist-test-slave-9cgk"
I20240501 08:10:42.028854 21944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "6877815dd0594c039522791c8671008e"
format_stamp: "Formatted at 2024-05-01 08:10:42 on dist-test-slave-9cgk"
I20240501 08:10:42.034147 21944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20240501 08:10:42.037907 21958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:42.039302 21944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 08:10:42.039517 21944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "6877815dd0594c039522791c8671008e"
format_stamp: "Formatted at 2024-05-01 08:10:42 on dist-test-slave-9cgk"
I20240501 08:10:42.039718 21944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:42.055049 21944 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:42.055871 21944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:42.056143 21944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:42.057101 21944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:42.058593 21944 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:42.058667 21944 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:42.058779 21944 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:42.058825 21944 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:42.088438 21944 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.129:36183
I20240501 08:10:42.088519 22070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.129:36183 every 8 connection(s)
I20240501 08:10:42.089879 21944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 08:10:42.096843 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 21944
I20240501 08:10:42.097034 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 08:10:42.100353 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.130:0
--local_ip_for_outbound_sockets=127.21.84.130
--webserver_interface=127.21.84.130
--webserver_port=0
--tserver_master_addrs=127.21.84.190:46653
--builtin_ntp_servers=127.21.84.148:36877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 08:10:42.102847 22071 heartbeater.cc:347] Connected to a master server at 127.21.84.190:46653
I20240501 08:10:42.103139 22071 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:42.103957 22071 heartbeater.cc:511] Master 127.21.84.190:46653 requested a full tablet report, sending...
I20240501 08:10:42.106154 21885 ts_manager.cc:194] Registered new tserver with Master: 6877815dd0594c039522791c8671008e (127.21.84.129:36183)
I20240501 08:10:42.108340 21885 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.129:47049
W20240501 08:10:42.424878 22075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:42.425176 22075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:42.425278 22075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:42.432672 22075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:42.432854 22075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.130
I20240501 08:10:42.442122 22075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:36877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.21.84.130
--webserver_port=0
--tserver_master_addrs=127.21.84.190:46653
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.130
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:42.443408 22075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:42.445164 22075 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:42.456859 22080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:42.457422 22081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:42.458181 22083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:42.458884 22075 server_base.cc:1034] running on GCE node
I20240501 08:10:42.459439 22075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:42.460280 22075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:42.461499 22075 hybrid_clock.cc:648] HybridClock initialized: now 1714551042461429 us; error 66 us; skew 500 ppm
I20240501 08:10:42.461819 22075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:42.463955 22075 webserver.cc:460] Webserver started at http://127.21.84.130:38215/ using document root <none> and password file <none>
I20240501 08:10:42.464653 22075 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:42.464778 22075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:42.465061 22075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:42.467644 22075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d51f882fe60341e6be26fe413eee4d24"
format_stamp: "Formatted at 2024-05-01 08:10:42 on dist-test-slave-9cgk"
I20240501 08:10:42.468688 22075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d51f882fe60341e6be26fe413eee4d24"
format_stamp: "Formatted at 2024-05-01 08:10:42 on dist-test-slave-9cgk"
I20240501 08:10:42.474062 22075 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 08:10:42.477373 22089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:42.478747 22075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:10:42.478881 22075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d51f882fe60341e6be26fe413eee4d24"
format_stamp: "Formatted at 2024-05-01 08:10:42 on dist-test-slave-9cgk"
I20240501 08:10:42.479034 22075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:42.511176 22075 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:42.511998 22075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:42.512290 22075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:42.513255 22075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:42.514863 22075 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:42.514941 22075 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:42.515069 22075 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:42.515122 22075 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:42.545825 22075 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.130:45147
I20240501 08:10:42.545866 22201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.130:45147 every 8 connection(s)
I20240501 08:10:42.547431 22075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 08:10:42.549579 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 22075
I20240501 08:10:42.549739 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 08:10:42.553191 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.131:0
--local_ip_for_outbound_sockets=127.21.84.131
--webserver_interface=127.21.84.131
--webserver_port=0
--tserver_master_addrs=127.21.84.190:46653
--builtin_ntp_servers=127.21.84.148:36877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 08:10:42.561302 22202 heartbeater.cc:347] Connected to a master server at 127.21.84.190:46653
I20240501 08:10:42.561655 22202 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:42.562674 22202 heartbeater.cc:511] Master 127.21.84.190:46653 requested a full tablet report, sending...
I20240501 08:10:42.564258 21885 ts_manager.cc:194] Registered new tserver with Master: d51f882fe60341e6be26fe413eee4d24 (127.21.84.130:45147)
I20240501 08:10:42.565315 21885 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.130:50717
W20240501 08:10:42.880488 22205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:42.880791 22205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:42.880887 22205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:42.888324 22205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:42.888497 22205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.131
I20240501 08:10:42.897466 22205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:36877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.21.84.131
--webserver_port=0
--tserver_master_addrs=127.21.84.190:46653
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.131
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:42.898799 22205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:42.900506 22205 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:42.912531 22211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:42.913120 22213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:42.913812 22210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:42.914577 22205 server_base.cc:1034] running on GCE node
I20240501 08:10:42.915213 22205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:42.916189 22205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:42.917392 22205 hybrid_clock.cc:648] HybridClock initialized: now 1714551042917337 us; error 46 us; skew 500 ppm
I20240501 08:10:42.917670 22205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:42.919931 22205 webserver.cc:460] Webserver started at http://127.21.84.131:46115/ using document root <none> and password file <none>
I20240501 08:10:42.920605 22205 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:42.920744 22205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:42.921062 22205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:42.923529 22205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "08c22cef467245e4ab35f10dd35e1bca"
format_stamp: "Formatted at 2024-05-01 08:10:42 on dist-test-slave-9cgk"
I20240501 08:10:42.924217 22205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "08c22cef467245e4ab35f10dd35e1bca"
format_stamp: "Formatted at 2024-05-01 08:10:42 on dist-test-slave-9cgk"
I20240501 08:10:42.929236 22205 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20240501 08:10:42.932528 22219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:42.933739 22205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 08:10:42.933883 22205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "08c22cef467245e4ab35f10dd35e1bca"
format_stamp: "Formatted at 2024-05-01 08:10:42 on dist-test-slave-9cgk"
I20240501 08:10:42.934095 22205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:42.952593 22205 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:42.953392 22205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:42.953671 22205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:42.954643 22205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:42.956058 22205 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:42.956132 22205 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:42.956248 22205 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:42.956297 22205 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:42.989988 22205 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.131:44741
I20240501 08:10:42.990041 22331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.131:44741 every 8 connection(s)
I20240501 08:10:42.991334 22205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 08:10:42.992336 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 22205
I20240501 08:10:42.992583 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 08:10:42.996084 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.132:0
--local_ip_for_outbound_sockets=127.21.84.132
--webserver_interface=127.21.84.132
--webserver_port=0
--tserver_master_addrs=127.21.84.190:46653
--builtin_ntp_servers=127.21.84.148:36877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 08:10:43.005954 22332 heartbeater.cc:347] Connected to a master server at 127.21.84.190:46653
I20240501 08:10:43.006263 22332 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:43.007041 22332 heartbeater.cc:511] Master 127.21.84.190:46653 requested a full tablet report, sending...
I20240501 08:10:43.008597 21885 ts_manager.cc:194] Registered new tserver with Master: 08c22cef467245e4ab35f10dd35e1bca (127.21.84.131:44741)
I20240501 08:10:43.009819 21885 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.131:35279
I20240501 08:10:43.112028 22071 heartbeater.cc:503] Master 127.21.84.190:46653 was elected leader, sending a full tablet report...
W20240501 08:10:43.317386 22335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:43.317694 22335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:43.317787 22335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:43.325045 22335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:43.325258 22335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.132
I20240501 08:10:43.334100 22335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:36877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.21.84.132
--webserver_port=0
--tserver_master_addrs=127.21.84.190:46653
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.132
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:43.335479 22335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:43.337139 22335 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:43.349484 22341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:43.349967 22343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:43.350850 22340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:43.351624 22335 server_base.cc:1034] running on GCE node
I20240501 08:10:43.352205 22335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:43.353132 22335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:43.354350 22335 hybrid_clock.cc:648] HybridClock initialized: now 1714551043354260 us; error 50 us; skew 500 ppm
I20240501 08:10:43.354665 22335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:43.356925 22335 webserver.cc:460] Webserver started at http://127.21.84.132:39705/ using document root <none> and password file <none>
I20240501 08:10:43.357666 22335 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:43.357805 22335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:43.358155 22335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:43.360764 22335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "d8c2ff546bdb4df693dc927f9423692e"
format_stamp: "Formatted at 2024-05-01 08:10:43 on dist-test-slave-9cgk"
I20240501 08:10:43.361515 22335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "d8c2ff546bdb4df693dc927f9423692e"
format_stamp: "Formatted at 2024-05-01 08:10:43 on dist-test-slave-9cgk"
I20240501 08:10:43.366748 22335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20240501 08:10:43.370126 22349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:43.371369 22335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20240501 08:10:43.371549 22335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "d8c2ff546bdb4df693dc927f9423692e"
format_stamp: "Formatted at 2024-05-01 08:10:43 on dist-test-slave-9cgk"
I20240501 08:10:43.371717 22335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:43.390962 22335 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:43.391819 22335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:43.392081 22335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:43.393028 22335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:43.394556 22335 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:43.394629 22335 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:43.394757 22335 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:43.394811 22335 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:43.425362 22335 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.132:44101
I20240501 08:10:43.425412 22461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.132:44101 every 8 connection(s)
I20240501 08:10:43.426658 22335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 08:10:43.435406 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 22335
I20240501 08:10:43.435570 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 08:10:43.438709 22462 heartbeater.cc:347] Connected to a master server at 127.21.84.190:46653
I20240501 08:10:43.438815 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.133:0
--local_ip_for_outbound_sockets=127.21.84.133
--webserver_interface=127.21.84.133
--webserver_port=0
--tserver_master_addrs=127.21.84.190:46653
--builtin_ntp_servers=127.21.84.148:36877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 08:10:43.439074 22462 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:43.439822 22462 heartbeater.cc:511] Master 127.21.84.190:46653 requested a full tablet report, sending...
I20240501 08:10:43.441210 21885 ts_manager.cc:194] Registered new tserver with Master: d8c2ff546bdb4df693dc927f9423692e (127.21.84.132:44101)
I20240501 08:10:43.442381 21885 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.132:46609
I20240501 08:10:43.568507 22202 heartbeater.cc:503] Master 127.21.84.190:46653 was elected leader, sending a full tablet report...
W20240501 08:10:43.771883 22465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:43.772188 22465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:43.772316 22465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:43.779855 22465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:43.780055 22465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.133
I20240501 08:10:43.789711 22465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:36877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.21.84.133
--webserver_port=0
--tserver_master_addrs=127.21.84.190:46653
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.133
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:43.791090 22465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:43.792938 22465 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:43.805717 22473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:43.806247 22470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:43.807025 22471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:43.807686 22465 server_base.cc:1034] running on GCE node
I20240501 08:10:43.808284 22465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:43.809216 22465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:43.810418 22465 hybrid_clock.cc:648] HybridClock initialized: now 1714551043810356 us; error 36 us; skew 500 ppm
I20240501 08:10:43.810710 22465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:43.812976 22465 webserver.cc:460] Webserver started at http://127.21.84.133:37249/ using document root <none> and password file <none>
I20240501 08:10:43.813720 22465 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:43.813848 22465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:43.814185 22465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:43.816876 22465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "626d80c517a04158a532a323ee9686e7"
format_stamp: "Formatted at 2024-05-01 08:10:43 on dist-test-slave-9cgk"
I20240501 08:10:43.817603 22465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "626d80c517a04158a532a323ee9686e7"
format_stamp: "Formatted at 2024-05-01 08:10:43 on dist-test-slave-9cgk"
I20240501 08:10:43.822733 22465 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 08:10:43.826130 22479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:43.827456 22465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 08:10:43.827636 22465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "626d80c517a04158a532a323ee9686e7"
format_stamp: "Formatted at 2024-05-01 08:10:43 on dist-test-slave-9cgk"
I20240501 08:10:43.827807 22465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:43.859767 22465 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:43.860682 22465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:43.860967 22465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:43.862469 22465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:43.863965 22465 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:43.864039 22465 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:43.864158 22465 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:43.864207 22465 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:43.898095 22465 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.133:35853
I20240501 08:10:43.898136 22591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.133:35853 every 8 connection(s)
I20240501 08:10:43.899359 22465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 08:10:43.906570 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 22465
I20240501 08:10:43.906778 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 08:10:43.912267 22592 heartbeater.cc:347] Connected to a master server at 127.21.84.190:46653
I20240501 08:10:43.912550 22592 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:43.913317 22592 heartbeater.cc:511] Master 127.21.84.190:46653 requested a full tablet report, sending...
I20240501 08:10:43.914613 21885 ts_manager.cc:194] Registered new tserver with Master: 626d80c517a04158a532a323ee9686e7 (127.21.84.133:35853)
I20240501 08:10:43.915364 21885 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.133:52587
I20240501 08:10:43.923204 21842 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 08:10:43.944396 21885 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49272:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 08:10:43.976716 22267 tablet_service.cc:1467] Processing CreateTablet for tablet 4c291ce646f2490c8101f8532fef1b31 (DEFAULT_TABLE table=TestTable [id=fbd0fccd30fa4736900707f14775e91e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:43.977349 22137 tablet_service.cc:1467] Processing CreateTablet for tablet 4c291ce646f2490c8101f8532fef1b31 (DEFAULT_TABLE table=TestTable [id=fbd0fccd30fa4736900707f14775e91e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:43.977349 22527 tablet_service.cc:1467] Processing CreateTablet for tablet 4c291ce646f2490c8101f8532fef1b31 (DEFAULT_TABLE table=TestTable [id=fbd0fccd30fa4736900707f14775e91e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:43.978066 22267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c291ce646f2490c8101f8532fef1b31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:43.978724 22137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c291ce646f2490c8101f8532fef1b31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:43.978744 22527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c291ce646f2490c8101f8532fef1b31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:43.992035 22611 tablet_bootstrap.cc:492] T 4c291ce646f2490c8101f8532fef1b31 P d51f882fe60341e6be26fe413eee4d24: Bootstrap starting.
I20240501 08:10:43.993891 22612 tablet_bootstrap.cc:492] T 4c291ce646f2490c8101f8532fef1b31 P 626d80c517a04158a532a323ee9686e7: Bootstrap starting.
I20240501 08:10:43.995679 22611 tablet_bootstrap.cc:654] T 4c291ce646f2490c8101f8532fef1b31 P d51f882fe60341e6be26fe413eee4d24: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:43.996853 22611 log.cc:824] T 4c291ce646f2490c8101f8532fef1b31 P d51f882fe60341e6be26fe413eee4d24: Log is configured to *not* fsync() on all Append() calls
I20240501 08:10:43.997133 22612 tablet_bootstrap.cc:654] T 4c291ce646f2490c8101f8532fef1b31 P 626d80c517a04158a532a323ee9686e7: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:43.998267 22612 log.cc:824] T 4c291ce646f2490c8101f8532fef1b31 P 626d80c517a04158a532a323ee9686e7: Log is configured to *not* fsync() on all Append() calls
I20240501 08:10:43.999646 22611 tablet_bootstrap.cc:492] T 4c291ce646f2490c8101f8532fef1b31 P d51f882fe60341e6be26fe413eee4d24: No bootstrap required, opened a new log
I20240501 08:10:43.999919 22611 ts_tablet_manager.cc:1367] T 4c291ce646f2490c8101f8532fef1b31 P d51f882fe60341e6be26fe413eee4d24: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240501 08:10:44.000264 22610 tablet_bootstrap.cc:492] T 4c291ce646f2490c8101f8532fef1b31 P 08c22cef467245e4ab35f10dd35e1bca: Bootstrap starting.
I20240501 08:10:44.000757 22612 tablet_bootstrap.cc:492] T 4c291ce646f2490c8101f8532fef1b31 P 626d80c517a04158a532a323ee9686e7: No bootstrap required, opened a new log
I20240501 08:10:44.000993 22612 ts_tablet_manager.cc:1367] T 4c291ce646f2490c8101f8532fef1b31 P 626d80c517a04158a532a323ee9686e7: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240501 08:10:44.004688 22610 tablet_bootstrap.cc:654] T 4c291ce646f2490c8101f8532fef1b31 P 08c22cef467245e4ab35f10dd35e1bca: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:44.005951 22610 log.cc:824] T 4c291ce646f2490c8101f8532fef1b31 P 08c22cef467245e4ab35f10dd35e1bca: Log is configured to *not* fsync() on all Append() calls
I20240501 08:10:44.008837 22610 tablet_bootstrap.cc:492] T 4c291ce646f2490c8101f8532fef1b31 P 08c22cef467245e4ab35f10dd35e1bca: No bootstrap required, opened a new log
I20240501 08:10:44.008765 22612 raft_consensus.cc:348] T 4c291ce646f2490c8101f8532fef1b31 P 626d80c517a04158a532a323ee9686e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c22cef467245e4ab35f10dd35e1bca" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 44741 } } peers { permanent_uuid: "626d80c517a04158a532a323ee9686e7" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 35853 } } peers { permanent_uuid: "d51f882fe60341e6be26fe413eee4d24" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 45147 } }
I20240501 08:10:44.009140 22610 ts_tablet_manager.cc:1367] T 4c291ce646f2490c8101f8532fef1b31 P 08c22cef467245e4ab35f10dd35e1bca: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.000s
I20240501 08:10:44.009130 22612 raft_consensus.cc:374] T 4c291ce646f2490c8101f8532fef1b31 P 626d80c517a04158a532a323ee9686e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:44.009235 22612 raft_consensus.cc:729] T 4c291ce646f2490c8101f8532fef1b31 P 626d80c517a04158a532a323ee9686e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 626d80c517a04158a532a323ee9686e7, State: Initialized, Role: FOLLOWER
I20240501 08:10:44.009078 22611 raft_consensus.cc:348] T 4c291ce646f2490c8101f8532fef1b31 P d51f882fe60341e6be26fe413eee4d24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c22cef467245e4ab35f10dd35e1bca" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 44741 } } peers { permanent_uuid: "626d80c517a04158a532a323ee9686e7" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 35853 } } peers { permanent_uuid: "d51f882fe60341e6be26fe413eee4d24" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 45147 } }
I20240501 08:10:44.009491 22611 raft_consensus.cc:374] T 4c291ce646f2490c8101f8532fef1b31 P d51f882fe60341e6be26fe413eee4d24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:44.009603 22611 raft_consensus.cc:729] T 4c291ce646f2490c8101f8532fef1b31 P d51f882fe60341e6be26fe413eee4d24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d51f882fe60341e6be26fe413eee4d24, State: Initialized, Role: FOLLOWER
I20240501 08:10:44.010179 22612 consensus_queue.cc:260] T 4c291ce646f2490c8101f8532fef1b31 P 626d80c517a04158a532a323ee9686e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c22cef467245e4ab35f10dd35e1bca" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 44741 } } peers { permanent_uuid: "626d80c517a04158a532a323ee9686e7" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 35853 } } peers { permanent_uuid: "d51f882fe60341e6be26fe413eee4d24" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 45147 } }
I20240501 08:10:44.010275 22611 consensus_queue.cc:260] T 4c291ce646f2490c8101f8532fef1b31 P d51f882fe60341e6be26fe413eee4d24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c22cef467245e4ab35f10dd35e1bca" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 44741 } } peers { permanent_uuid: "626d80c517a04158a532a323ee9686e7" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 35853 } } peers { permanent_uuid: "d51f882fe60341e6be26fe413eee4d24" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 45147 } }
I20240501 08:10:44.012352 22332 heartbeater.cc:503] Master 127.21.84.190:46653 was elected leader, sending a full tablet report...
I20240501 08:10:44.021168 22610 raft_consensus.cc:348] T 4c291ce646f2490c8101f8532fef1b31 P 08c22cef467245e4ab35f10dd35e1bca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c22cef467245e4ab35f10dd35e1bca" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 44741 } } peers { permanent_uuid: "626d80c517a04158a532a323ee9686e7" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 35853 } } peers { permanent_uuid: "d51f882fe60341e6be26fe413eee4d24" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 45147 } }
I20240501 08:10:44.022809 22610 raft_consensus.cc:374] T 4c291ce646f2490c8101f8532fef1b31 P 08c22cef467245e4ab35f10dd35e1bca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:44.022996 22610 raft_consensus.cc:729] T 4c291ce646f2490c8101f8532fef1b31 P 08c22cef467245e4ab35f10dd35e1bca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08c22cef467245e4ab35f10dd35e1bca, State: Initialized, Role: FOLLOWER
I20240501 08:10:44.023535 22612 ts_tablet_manager.cc:1398] T 4c291ce646f2490c8101f8532fef1b31 P 626d80c517a04158a532a323ee9686e7: Time spent starting tablet: real 0.022s user 0.015s sys 0.008s
I20240501 08:10:44.023874 22611 ts_tablet_manager.cc:1398] T 4c291ce646f2490c8101f8532fef1b31 P d51f882fe60341e6be26fe413eee4d24: Time spent starting tablet: real 0.024s user 0.006s sys 0.016s
I20240501 08:10:44.024327 22610 consensus_queue.cc:260] T 4c291ce646f2490c8101f8532fef1b31 P 08c22cef467245e4ab35f10dd35e1bca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c22cef467245e4ab35f10dd35e1bca" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 44741 } } peers { permanent_uuid: "626d80c517a04158a532a323ee9686e7" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 35853 } } peers { permanent_uuid: "d51f882fe60341e6be26fe413eee4d24" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 45147 } }
I20240501 08:10:44.024726 22592 heartbeater.cc:503] Master 127.21.84.190:46653 was elected leader, sending a full tablet report...
I20240501 08:10:44.026078 22610 ts_tablet_manager.cc:1398] T 4c291ce646f2490c8101f8532fef1b31 P 08c22cef467245e4ab35f10dd35e1bca: Time spent starting tablet: real 0.017s user 0.013s sys 0.005s
W20240501 08:10:44.050100 22203 tablet.cc:2369] T 4c291ce646f2490c8101f8532fef1b31 P d51f882fe60341e6be26fe413eee4d24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:10:44.092422 22616 raft_consensus.cc:482] T 4c291ce646f2490c8101f8532fef1b31 P 626d80c517a04158a532a323ee9686e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:44.092695 22616 raft_consensus.cc:504] T 4c291ce646f2490c8101f8532fef1b31 P 626d80c517a04158a532a323ee9686e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c22cef467245e4ab35f10dd35e1bca" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 44741 } } peers { permanent_uuid: "626d80c517a04158a532a323ee9686e7" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 35853 } } peers { permanent_uuid: "d51f882fe60341e6be26fe413eee4d24" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 45147 } }
I20240501 08:10:44.094523 22616 leader_election.cc:288] T 4c291ce646f2490c8101f8532fef1b31 P 626d80c517a04158a532a323ee9686e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08c22cef467245e4ab35f10dd35e1bca (127.21.84.131:44741), d51f882fe60341e6be26fe413eee4d24 (127.21.84.130:45147)
I20240501 08:10:44.100960 22287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c291ce646f2490c8101f8532fef1b31" candidate_uuid: "626d80c517a04158a532a323ee9686e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08c22cef467245e4ab35f10dd35e1bca" is_pre_election: true
I20240501 08:10:44.101464 22287 raft_consensus.cc:2454] T 4c291ce646f2490c8101f8532fef1b31 P 08c22cef467245e4ab35f10dd35e1bca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 626d80c517a04158a532a323ee9686e7 in term 0.
I20240501 08:10:44.102010 22157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c291ce646f2490c8101f8532fef1b31" candidate_uuid: "626d80c517a04158a532a323ee9686e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d51f882fe60341e6be26fe413eee4d24" is_pre_election: true
I20240501 08:10:44.102120 22483 leader_election.cc:302] T 4c291ce646f2490c8101f8532fef1b31 P 626d80c517a04158a532a323ee9686e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08c22cef467245e4ab35f10dd35e1bca, 626d80c517a04158a532a323ee9686e7; no voters:
I20240501 08:10:44.102491 22157 raft_consensus.cc:2454] T 4c291ce646f2490c8101f8532fef1b31 P d51f882fe60341e6be26fe413eee4d24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 626d80c517a04158a532a323ee9686e7 in term 0.
I20240501 08:10:44.102514 22616 raft_consensus.cc:2781] T 4c291ce646f2490c8101f8532fef1b31 P 626d80c517a04158a532a323ee9686e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:44.102646 22616 raft_consensus.cc:482] T 4c291ce646f2490c8101f8532fef1b31 P 626d80c517a04158a532a323ee9686e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:44.102738 22616 raft_consensus.cc:3037] T 4c291ce646f2490c8101f8532fef1b31 P 626d80c517a04158a532a323ee9686e7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:44.104775 22616 raft_consensus.cc:504] T 4c291ce646f2490c8101f8532fef1b31 P 626d80c517a04158a532a323ee9686e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c22cef467245e4ab35f10dd35e1bca" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 44741 } } peers { permanent_uuid: "626d80c517a04158a532a323ee9686e7" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 35853 } } peers { permanent_uuid: "d51f882fe60341e6be26fe413eee4d24" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 45147 } }
I20240501 08:10:44.105301 22616 leader_election.cc:288] T 4c291ce646f2490c8101f8532fef1b31 P 626d80c517a04158a532a323ee9686e7 [CANDIDATE]: Term 1 election: Requested vote from peers 08c22cef467245e4ab35f10dd35e1bca (127.21.84.131:44741), d51f882fe60341e6be26fe413eee4d24 (127.21.84.130:45147)
I20240501 08:10:44.105710 22287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c291ce646f2490c8101f8532fef1b31" candidate_uuid: "626d80c517a04158a532a323ee9686e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08c22cef467245e4ab35f10dd35e1bca"
I20240501 08:10:44.105783 22157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c291ce646f2490c8101f8532fef1b31" candidate_uuid: "626d80c517a04158a532a323ee9686e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d51f882fe60341e6be26fe413eee4d24"
I20240501 08:10:44.105880 22287 raft_consensus.cc:3037] T 4c291ce646f2490c8101f8532fef1b31 P 08c22cef467245e4ab35f10dd35e1bca [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:44.105996 22157 raft_consensus.cc:3037] T 4c291ce646f2490c8101f8532fef1b31 P d51f882fe60341e6be26fe413eee4d24 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:44.108036 22157 raft_consensus.cc:2454] T 4c291ce646f2490c8101f8532fef1b31 P d51f882fe60341e6be26fe413eee4d24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 626d80c517a04158a532a323ee9686e7 in term 1.
I20240501 08:10:44.108382 22287 raft_consensus.cc:2454] T 4c291ce646f2490c8101f8532fef1b31 P 08c22cef467245e4ab35f10dd35e1bca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 626d80c517a04158a532a323ee9686e7 in term 1.
I20240501 08:10:44.108419 22483 leader_election.cc:302] T 4c291ce646f2490c8101f8532fef1b31 P 626d80c517a04158a532a323ee9686e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 626d80c517a04158a532a323ee9686e7, d51f882fe60341e6be26fe413eee4d24; no voters:
I20240501 08:10:44.108711 22616 raft_consensus.cc:2781] T 4c291ce646f2490c8101f8532fef1b31 P 626d80c517a04158a532a323ee9686e7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:44.109118 22616 raft_consensus.cc:686] T 4c291ce646f2490c8101f8532fef1b31 P 626d80c517a04158a532a323ee9686e7 [term 1 LEADER]: Becoming Leader. State: Replica: 626d80c517a04158a532a323ee9686e7, State: Running, Role: LEADER
I20240501 08:10:44.109563 22616 consensus_queue.cc:237] T 4c291ce646f2490c8101f8532fef1b31 P 626d80c517a04158a532a323ee9686e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c22cef467245e4ab35f10dd35e1bca" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 44741 } } peers { permanent_uuid: "626d80c517a04158a532a323ee9686e7" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 35853 } } peers { permanent_uuid: "d51f882fe60341e6be26fe413eee4d24" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 45147 } }
I20240501 08:10:44.114784 21884 catalog_manager.cc:5484] T 4c291ce646f2490c8101f8532fef1b31 P 626d80c517a04158a532a323ee9686e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 626d80c517a04158a532a323ee9686e7 (127.21.84.133). New cstate: current_term: 1 leader_uuid: "626d80c517a04158a532a323ee9686e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08c22cef467245e4ab35f10dd35e1bca" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 44741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "626d80c517a04158a532a323ee9686e7" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 35853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d51f882fe60341e6be26fe413eee4d24" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 45147 } health_report { overall_health: UNKNOWN } } }
I20240501 08:10:44.133091 21842 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 08:10:44.135744 21842 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6877815dd0594c039522791c8671008e to finish bootstrapping
I20240501 08:10:44.145646 21842 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d51f882fe60341e6be26fe413eee4d24 to finish bootstrapping
W20240501 08:10:44.151006 22593 tablet.cc:2369] T 4c291ce646f2490c8101f8532fef1b31 P 626d80c517a04158a532a323ee9686e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:10:44.153669 21842 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 08c22cef467245e4ab35f10dd35e1bca to finish bootstrapping
I20240501 08:10:44.161311 21842 ts_itest-base.cc:233] Waiting for 0 tablets on tserver d8c2ff546bdb4df693dc927f9423692e to finish bootstrapping
I20240501 08:10:44.170158 21842 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 626d80c517a04158a532a323ee9686e7 to finish bootstrapping
W20240501 08:10:44.243780 22333 tablet.cc:2369] T 4c291ce646f2490c8101f8532fef1b31 P 08c22cef467245e4ab35f10dd35e1bca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:10:44.445168 22462 heartbeater.cc:503] Master 127.21.84.190:46653 was elected leader, sending a full tablet report...
I20240501 08:10:44.509829 22616 consensus_queue.cc:1035] T 4c291ce646f2490c8101f8532fef1b31 P 626d80c517a04158a532a323ee9686e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08c22cef467245e4ab35f10dd35e1bca" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 44741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:44.519261 22616 consensus_queue.cc:1035] T 4c291ce646f2490c8101f8532fef1b31 P 626d80c517a04158a532a323ee9686e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d51f882fe60341e6be26fe413eee4d24" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:44.681133 22267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:44.692180 22006 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:44.692567 22527 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:10:44.699342 22137 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:44.717272 22397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:45.322134 22527 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:10:45.329313 22397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:45.333786 22006 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:45.333911 22137 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:45.335697 22267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:45.503033 22006 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:45.503151 22527 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:10:45.507592 22267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:45.519078 22137 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:45.521154 22397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:45.718837 22267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:45.728518 22137 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:45.729112 22006 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:45.730998 22397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:45.733698 22527 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:10:46.284149 21885 ts_manager.cc:292] Set tserver state for 6877815dd0594c039522791c8671008e to MAINTENANCE_MODE
I20240501 08:10:47.791040 22527 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:10:47.812180 22267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:47.827013 22006 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:47.833787 22137 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:47.840340 22397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:48.030037 22267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:48.031198 22006 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:48.033067 22527 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:10:48.042735 22397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:48.045377 22137 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:48.229877 22267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:48.232113 22527 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:10:48.242769 22006 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:48.245074 22137 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:48.245684 22397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:48.392946 22267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:48.401878 22006 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:48.402060 22397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:48.404919 22527 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:10:48.414716 22137 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:48.575073 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 22075
I20240501 08:10:48.576546 22196 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:10:48.677628 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 22075
W20240501 08:10:48.743885 22483 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.84.130:45147: connect: Connection refused (error 111)
W20240501 08:10:48.745676 22483 consensus_peers.cc:485] T 4c291ce646f2490c8101f8532fef1b31 P 626d80c517a04158a532a323ee9686e7 -> Peer d51f882fe60341e6be26fe413eee4d24 (127.21.84.130:45147): Couldn't send request to peer d51f882fe60341e6be26fe413eee4d24. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.130:45147: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:10:49.157699 22267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:49.164160 22527 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:10:49.168792 22397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:49.170439 22006 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:49.299506 22267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:49.302798 22006 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:49.306602 22527 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:10:49.315912 22397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:49.489393 22267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:49.493774 22527 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:10:49.499351 22006 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:49.499351 22397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:49.635684 22267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:49.639823 22527 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:10:49.641826 22397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:49.642328 22006 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:49.796084 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 21944
I20240501 08:10:49.797508 22065 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:10:49.904606 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 21944
I20240501 08:10:49.924454 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 22205
I20240501 08:10:49.925514 22327 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:10:50.034016 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 22205
I20240501 08:10:50.049753 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 22335
I20240501 08:10:50.050820 22457 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:10:50.154173 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 22335
I20240501 08:10:50.173514 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 22465
I20240501 08:10:50.174566 22587 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:10:50.283331 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 22465
I20240501 08:10:50.303017 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 21854
I20240501 08:10:50.304183 21915 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:10:50.432631 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 21854
2024-05-01T08:10:50Z chronyd exiting
[ OK ] RebalanceIgnoredTserversTest.Basic (9387 ms)
[----------] 1 test from RebalanceIgnoredTserversTest (9387 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/1
I20240501 08:10:50.478194 21842 test_util.cc:268] Using random seed: 105914954
I20240501 08:10:50.479943 21842 ts_itest-base.cc:112] Starting cluster with:
I20240501 08:10:50.480028 21842 ts_itest-base.cc:113] --------------
I20240501 08:10:50.480065 21842 ts_itest-base.cc:114] 5 tablet servers
I20240501 08:10:50.480116 21842 ts_itest-base.cc:115] 3 replicas per TS
I20240501 08:10:50.480156 21842 ts_itest-base.cc:116] --------------
2024-05-01T08:10:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T08:10:50Z Disabled control of system clock
I20240501 08:10:50.503430 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.84.190:38497
--webserver_interface=127.21.84.190
--webserver_port=0
--builtin_ntp_servers=127.21.84.148:46539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.84.190:38497
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 08:10:50.831019 23067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:50.831316 23067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:50.831411 23067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:50.838924 23067 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:10:50.839043 23067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 08:10:50.839093 23067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:50.839150 23067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 08:10:50.839197 23067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 08:10:50.848299 23067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:46539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.21.84.190:38497
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.84.190:38497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.21.84.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:50.849625 23067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:50.851450 23067 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:50.862646 23075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:50.863301 23073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:50.863950 23072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:50.864718 23067 server_base.cc:1034] running on GCE node
I20240501 08:10:50.865708 23067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:50.867487 23067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:50.868697 23067 hybrid_clock.cc:648] HybridClock initialized: now 1714551050868638 us; error 50 us; skew 500 ppm
I20240501 08:10:50.869078 23067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:50.871647 23067 webserver.cc:460] Webserver started at http://127.21.84.190:35821/ using document root <none> and password file <none>
I20240501 08:10:50.872393 23067 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:50.872505 23067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:50.872859 23067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:50.875794 23067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "a6bc798e255c4871a0697f174c239e9a"
format_stamp: "Formatted at 2024-05-01 08:10:50 on dist-test-slave-9cgk"
I20240501 08:10:50.876685 23067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "a6bc798e255c4871a0697f174c239e9a"
format_stamp: "Formatted at 2024-05-01 08:10:50 on dist-test-slave-9cgk"
I20240501 08:10:50.882755 23067 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20240501 08:10:50.886116 23081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:50.887748 23067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 08:10:50.887941 23067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/wal
uuid: "a6bc798e255c4871a0697f174c239e9a"
format_stamp: "Formatted at 2024-05-01 08:10:50 on dist-test-slave-9cgk"
I20240501 08:10:50.888178 23067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:50.904376 23067 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:50.905236 23067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:50.905470 23067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:50.922959 23067 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.190:38497
I20240501 08:10:50.922978 23132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.190:38497 every 8 connection(s)
I20240501 08:10:50.924288 23067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 08:10:50.928103 23133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:50.932049 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 23067
I20240501 08:10:50.932243 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 08:10:50.938305 23133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6bc798e255c4871a0697f174c239e9a: Bootstrap starting.
I20240501 08:10:50.942170 23133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6bc798e255c4871a0697f174c239e9a: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:50.943171 23133 log.cc:824] T 00000000000000000000000000000000 P a6bc798e255c4871a0697f174c239e9a: Log is configured to *not* fsync() on all Append() calls
I20240501 08:10:50.946362 23133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6bc798e255c4871a0697f174c239e9a: No bootstrap required, opened a new log
I20240501 08:10:50.952842 23133 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6bc798e255c4871a0697f174c239e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6bc798e255c4871a0697f174c239e9a" member_type: VOTER last_known_addr { host: "127.21.84.190" port: 38497 } }
I20240501 08:10:50.953197 23133 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6bc798e255c4871a0697f174c239e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:50.953291 23133 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6bc798e255c4871a0697f174c239e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6bc798e255c4871a0697f174c239e9a, State: Initialized, Role: FOLLOWER
I20240501 08:10:50.953858 23133 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6bc798e255c4871a0697f174c239e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6bc798e255c4871a0697f174c239e9a" member_type: VOTER last_known_addr { host: "127.21.84.190" port: 38497 } }
I20240501 08:10:50.954084 23133 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6bc798e255c4871a0697f174c239e9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:10:50.954205 23133 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6bc798e255c4871a0697f174c239e9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:10:50.954347 23133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6bc798e255c4871a0697f174c239e9a [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:50.990280 23133 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6bc798e255c4871a0697f174c239e9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6bc798e255c4871a0697f174c239e9a" member_type: VOTER last_known_addr { host: "127.21.84.190" port: 38497 } }
I20240501 08:10:50.990957 23133 leader_election.cc:302] T 00000000000000000000000000000000 P a6bc798e255c4871a0697f174c239e9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6bc798e255c4871a0697f174c239e9a; no voters:
I20240501 08:10:50.991616 23133 leader_election.cc:288] T 00000000000000000000000000000000 P a6bc798e255c4871a0697f174c239e9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:10:50.991796 23138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6bc798e255c4871a0697f174c239e9a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:50.992894 23138 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6bc798e255c4871a0697f174c239e9a [term 1 LEADER]: Becoming Leader. State: Replica: a6bc798e255c4871a0697f174c239e9a, State: Running, Role: LEADER
I20240501 08:10:50.993597 23138 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6bc798e255c4871a0697f174c239e9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6bc798e255c4871a0697f174c239e9a" member_type: VOTER last_known_addr { host: "127.21.84.190" port: 38497 } }
I20240501 08:10:50.993925 23133 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6bc798e255c4871a0697f174c239e9a [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:10:50.996824 23139 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6bc798e255c4871a0697f174c239e9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6bc798e255c4871a0697f174c239e9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6bc798e255c4871a0697f174c239e9a" member_type: VOTER last_known_addr { host: "127.21.84.190" port: 38497 } } }
I20240501 08:10:50.997350 23140 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6bc798e255c4871a0697f174c239e9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6bc798e255c4871a0697f174c239e9a. Latest consensus state: current_term: 1 leader_uuid: "a6bc798e255c4871a0697f174c239e9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6bc798e255c4871a0697f174c239e9a" member_type: VOTER last_known_addr { host: "127.21.84.190" port: 38497 } } }
I20240501 08:10:50.997538 23139 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6bc798e255c4871a0697f174c239e9a [sys.catalog]: This master's current role is: LEADER
I20240501 08:10:50.997609 23140 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6bc798e255c4871a0697f174c239e9a [sys.catalog]: This master's current role is: LEADER
I20240501 08:10:50.999254 23147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:10:51.004194 23147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:10:51.012476 23147 catalog_manager.cc:1347] Generated new cluster ID: ec9d8d3fbc83476fb40cf72c455d2162
I20240501 08:10:51.012586 23147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:10:51.067514 23147 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:10:51.068725 23147 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:10:51.085809 23147 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a6bc798e255c4871a0697f174c239e9a: Generated new TSK 0
I20240501 08:10:51.086866 23147 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:10:51.105857 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.129:0
--local_ip_for_outbound_sockets=127.21.84.129
--webserver_interface=127.21.84.129
--webserver_port=0
--tserver_master_addrs=127.21.84.190:38497
--builtin_ntp_servers=127.21.84.148:46539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 08:10:51.466418 23157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:51.466742 23157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:51.466871 23157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:51.474305 23157 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:10:51.474436 23157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:51.474591 23157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.129
I20240501 08:10:51.483676 23157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:46539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.21.84.129
--webserver_port=0
--tserver_master_addrs=127.21.84.190:38497
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.129
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:51.485002 23157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:51.486755 23157 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:51.499286 23162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:51.500097 23163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:51.500871 23165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:51.501667 23157 server_base.cc:1034] running on GCE node
I20240501 08:10:51.502311 23157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:51.503279 23157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:51.504475 23157 hybrid_clock.cc:648] HybridClock initialized: now 1714551051504426 us; error 47 us; skew 500 ppm
I20240501 08:10:51.504805 23157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:51.507360 23157 webserver.cc:460] Webserver started at http://127.21.84.129:42177/ using document root <none> and password file <none>
I20240501 08:10:51.508118 23157 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:51.508234 23157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:51.508564 23157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:51.511282 23157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "06d814f013b240af85b4209cb43b0cd7"
format_stamp: "Formatted at 2024-05-01 08:10:51 on dist-test-slave-9cgk"
I20240501 08:10:51.512055 23157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "06d814f013b240af85b4209cb43b0cd7"
format_stamp: "Formatted at 2024-05-01 08:10:51 on dist-test-slave-9cgk"
I20240501 08:10:51.517410 23157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20240501 08:10:51.520918 23171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:51.522281 23157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:10:51.522437 23157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "06d814f013b240af85b4209cb43b0cd7"
format_stamp: "Formatted at 2024-05-01 08:10:51 on dist-test-slave-9cgk"
I20240501 08:10:51.522642 23157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:51.541563 23157 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:51.542510 23157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:51.542785 23157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:51.543749 23157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:51.545269 23157 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:51.545365 23157 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:51.545470 23157 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:51.545537 23157 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:51.577731 23157 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.129:39167
I20240501 08:10:51.577785 23283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.129:39167 every 8 connection(s)
I20240501 08:10:51.579139 23157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 08:10:51.586763 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 23157
I20240501 08:10:51.586954 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 08:10:51.590041 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.130:0
--local_ip_for_outbound_sockets=127.21.84.130
--webserver_interface=127.21.84.130
--webserver_port=0
--tserver_master_addrs=127.21.84.190:38497
--builtin_ntp_servers=127.21.84.148:46539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:10:51.592542 23284 heartbeater.cc:347] Connected to a master server at 127.21.84.190:38497
I20240501 08:10:51.592926 23284 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:51.594070 23284 heartbeater.cc:511] Master 127.21.84.190:38497 requested a full tablet report, sending...
I20240501 08:10:51.596230 23098 ts_manager.cc:194] Registered new tserver with Master: 06d814f013b240af85b4209cb43b0cd7 (127.21.84.129:39167)
I20240501 08:10:51.597676 23098 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.129:39043
W20240501 08:10:51.915158 23288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:51.915488 23288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:51.915621 23288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:51.923149 23288 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:10:51.923314 23288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:51.923483 23288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.130
I20240501 08:10:51.932852 23288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:46539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.21.84.130
--webserver_port=0
--tserver_master_addrs=127.21.84.190:38497
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.130
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:51.934226 23288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:51.935951 23288 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:51.947978 23293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:51.948596 23294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:51.949512 23296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:51.950352 23288 server_base.cc:1034] running on GCE node
I20240501 08:10:51.950971 23288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:51.951918 23288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:51.953127 23288 hybrid_clock.cc:648] HybridClock initialized: now 1714551051953069 us; error 40 us; skew 500 ppm
I20240501 08:10:51.953550 23288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:51.955834 23288 webserver.cc:460] Webserver started at http://127.21.84.130:38081/ using document root <none> and password file <none>
I20240501 08:10:51.956574 23288 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:51.956686 23288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:51.957031 23288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:51.959669 23288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "acc1d95756c441879c7faa6760d0dfc9"
format_stamp: "Formatted at 2024-05-01 08:10:51 on dist-test-slave-9cgk"
I20240501 08:10:51.960443 23288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "acc1d95756c441879c7faa6760d0dfc9"
format_stamp: "Formatted at 2024-05-01 08:10:51 on dist-test-slave-9cgk"
I20240501 08:10:51.965739 23288 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20240501 08:10:51.969134 23302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:51.970485 23288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:10:51.970641 23288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "acc1d95756c441879c7faa6760d0dfc9"
format_stamp: "Formatted at 2024-05-01 08:10:51 on dist-test-slave-9cgk"
I20240501 08:10:51.970844 23288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:51.994572 23288 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:51.995599 23288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:51.995888 23288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:51.996843 23288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:51.998404 23288 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:51.998504 23288 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:51.998625 23288 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:51.998694 23288 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:52.033171 23288 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.130:44483
I20240501 08:10:52.033231 23414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.130:44483 every 8 connection(s)
I20240501 08:10:52.034782 23288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 08:10:52.038939 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 23288
I20240501 08:10:52.039167 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 08:10:52.042838 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.131:0
--local_ip_for_outbound_sockets=127.21.84.131
--webserver_interface=127.21.84.131
--webserver_port=0
--tserver_master_addrs=127.21.84.190:38497
--builtin_ntp_servers=127.21.84.148:46539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:10:52.048995 23415 heartbeater.cc:347] Connected to a master server at 127.21.84.190:38497
I20240501 08:10:52.049314 23415 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:52.050390 23415 heartbeater.cc:511] Master 127.21.84.190:38497 requested a full tablet report, sending...
I20240501 08:10:52.052107 23098 ts_manager.cc:194] Registered new tserver with Master: acc1d95756c441879c7faa6760d0dfc9 (127.21.84.130:44483)
I20240501 08:10:52.052919 23098 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.130:42603
W20240501 08:10:52.367744 23418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:52.368072 23418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:52.368227 23418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:52.375941 23418 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:10:52.376093 23418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:52.376247 23418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.131
I20240501 08:10:52.385509 23418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:46539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.21.84.131
--webserver_port=0
--tserver_master_addrs=127.21.84.190:38497
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.131
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:52.386952 23418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:52.388751 23418 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:52.401261 23423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:52.402192 23418 server_base.cc:1034] running on GCE node
W20240501 08:10:52.402823 23426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:52.403484 23424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:52.404158 23418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:52.405092 23418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:52.406281 23418 hybrid_clock.cc:648] HybridClock initialized: now 1714551052406225 us; error 56 us; skew 500 ppm
I20240501 08:10:52.406590 23418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:52.408730 23418 webserver.cc:460] Webserver started at http://127.21.84.131:42923/ using document root <none> and password file <none>
I20240501 08:10:52.409480 23418 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:52.409605 23418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:52.409919 23418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:52.412678 23418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "9c47bb4ff8cc4b2b9311b9810f9c584f"
format_stamp: "Formatted at 2024-05-01 08:10:52 on dist-test-slave-9cgk"
I20240501 08:10:52.413424 23418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "9c47bb4ff8cc4b2b9311b9810f9c584f"
format_stamp: "Formatted at 2024-05-01 08:10:52 on dist-test-slave-9cgk"
I20240501 08:10:52.418849 23418 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20240501 08:10:52.422982 23432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:52.424340 23418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 08:10:52.424702 23418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9c47bb4ff8cc4b2b9311b9810f9c584f"
format_stamp: "Formatted at 2024-05-01 08:10:52 on dist-test-slave-9cgk"
I20240501 08:10:52.424939 23418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:52.456221 23418 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:52.457157 23418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:52.457434 23418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:52.458398 23418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:52.459928 23418 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:52.460024 23418 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:52.460148 23418 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:52.460217 23418 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:52.492092 23418 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.131:45969
I20240501 08:10:52.492173 23544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.131:45969 every 8 connection(s)
I20240501 08:10:52.493587 23418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 08:10:52.502298 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 23418
I20240501 08:10:52.502489 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 08:10:52.505875 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.132:0
--local_ip_for_outbound_sockets=127.21.84.132
--webserver_interface=127.21.84.132
--webserver_port=0
--tserver_master_addrs=127.21.84.190:38497
--builtin_ntp_servers=127.21.84.148:46539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:10:52.507000 23545 heartbeater.cc:347] Connected to a master server at 127.21.84.190:38497
I20240501 08:10:52.507390 23545 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:52.508538 23545 heartbeater.cc:511] Master 127.21.84.190:38497 requested a full tablet report, sending...
I20240501 08:10:52.510223 23098 ts_manager.cc:194] Registered new tserver with Master: 9c47bb4ff8cc4b2b9311b9810f9c584f (127.21.84.131:45969)
I20240501 08:10:52.511408 23098 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.131:48595
I20240501 08:10:52.600561 23284 heartbeater.cc:503] Master 127.21.84.190:38497 was elected leader, sending a full tablet report...
W20240501 08:10:52.841162 23548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:52.841605 23548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:52.841765 23548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:52.855772 23548 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:10:52.855959 23548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:52.856196 23548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.132
I20240501 08:10:52.873441 23548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:46539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.21.84.132
--webserver_port=0
--tserver_master_addrs=127.21.84.190:38497
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.132
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:52.875358 23548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:52.877768 23548 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:52.895723 23553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:52.896339 23554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:52.897178 23556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:52.898361 23548 server_base.cc:1034] running on GCE node
I20240501 08:10:52.899262 23548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:52.900594 23548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:52.901857 23548 hybrid_clock.cc:648] HybridClock initialized: now 1714551052901752 us; error 85 us; skew 500 ppm
I20240501 08:10:52.902334 23548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:52.910959 23548 webserver.cc:460] Webserver started at http://127.21.84.132:44783/ using document root <none> and password file <none>
I20240501 08:10:52.913098 23548 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:52.913321 23548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:52.913836 23548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:52.918306 23548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "b17af124243a4af3bf8a430803a498e0"
format_stamp: "Formatted at 2024-05-01 08:10:52 on dist-test-slave-9cgk"
I20240501 08:10:52.919448 23548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "b17af124243a4af3bf8a430803a498e0"
format_stamp: "Formatted at 2024-05-01 08:10:52 on dist-test-slave-9cgk"
I20240501 08:10:52.927675 23548 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20240501 08:10:52.932925 23562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:52.934827 23548 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240501 08:10:52.935017 23548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "b17af124243a4af3bf8a430803a498e0"
format_stamp: "Formatted at 2024-05-01 08:10:52 on dist-test-slave-9cgk"
I20240501 08:10:52.935305 23548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:52.969058 23548 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:52.970343 23548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:52.970748 23548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:52.972108 23548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:52.974400 23548 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:52.974519 23548 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:52.974748 23548 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:52.974896 23548 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:53.018849 23548 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.132:35337
I20240501 08:10:53.018884 23675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.132:35337 every 8 connection(s)
I20240501 08:10:53.020133 23548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 08:10:53.025436 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 23548
I20240501 08:10:53.025631 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 08:10:53.028765 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.133:0
--local_ip_for_outbound_sockets=127.21.84.133
--webserver_interface=127.21.84.133
--webserver_port=0
--tserver_master_addrs=127.21.84.190:38497
--builtin_ntp_servers=127.21.84.148:46539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:10:53.036659 23676 heartbeater.cc:347] Connected to a master server at 127.21.84.190:38497
I20240501 08:10:53.037003 23676 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:53.037840 23676 heartbeater.cc:511] Master 127.21.84.190:38497 requested a full tablet report, sending...
I20240501 08:10:53.039295 23098 ts_manager.cc:194] Registered new tserver with Master: b17af124243a4af3bf8a430803a498e0 (127.21.84.132:35337)
I20240501 08:10:53.040057 23098 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.132:48779
I20240501 08:10:53.055860 23415 heartbeater.cc:503] Master 127.21.84.190:38497 was elected leader, sending a full tablet report...
W20240501 08:10:53.370517 23680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:53.370848 23680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:53.370939 23680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:53.378578 23680 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:10:53.378717 23680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:53.378882 23680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.133
I20240501 08:10:53.388242 23680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:46539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.21.84.133
--webserver_port=0
--tserver_master_addrs=127.21.84.190:38497
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.133
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:53.389842 23680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:53.391762 23680 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:53.403946 23685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:53.404836 23686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:53.405445 23688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:53.406303 23680 server_base.cc:1034] running on GCE node
I20240501 08:10:53.406917 23680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:53.407855 23680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:53.409036 23680 hybrid_clock.cc:648] HybridClock initialized: now 1714551053408982 us; error 51 us; skew 500 ppm
I20240501 08:10:53.409333 23680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:53.411665 23680 webserver.cc:460] Webserver started at http://127.21.84.133:33505/ using document root <none> and password file <none>
I20240501 08:10:53.412380 23680 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:53.412482 23680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:53.412799 23680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:53.415366 23680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "248cfd6a113e436abf83f58c7800ddf5"
format_stamp: "Formatted at 2024-05-01 08:10:53 on dist-test-slave-9cgk"
I20240501 08:10:53.416111 23680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "248cfd6a113e436abf83f58c7800ddf5"
format_stamp: "Formatted at 2024-05-01 08:10:53 on dist-test-slave-9cgk"
I20240501 08:10:53.421587 23680 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240501 08:10:53.424940 23694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:53.426231 23680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 08:10:53.426383 23680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "248cfd6a113e436abf83f58c7800ddf5"
format_stamp: "Formatted at 2024-05-01 08:10:53 on dist-test-slave-9cgk"
I20240501 08:10:53.426579 23680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:53.446643 23680 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:53.447517 23680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:53.447793 23680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:53.448776 23680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:53.450277 23680 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:53.450362 23680 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:53.450469 23680 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:53.450551 23680 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:53.485756 23680 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.133:42509
I20240501 08:10:53.485806 23806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.133:42509 every 8 connection(s)
I20240501 08:10:53.487169 23680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 08:10:53.490447 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 23680
I20240501 08:10:53.490584 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 08:10:53.500341 23807 heartbeater.cc:347] Connected to a master server at 127.21.84.190:38497
I20240501 08:10:53.500620 23807 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:53.501387 23807 heartbeater.cc:511] Master 127.21.84.190:38497 requested a full tablet report, sending...
I20240501 08:10:53.502728 23098 ts_manager.cc:194] Registered new tserver with Master: 248cfd6a113e436abf83f58c7800ddf5 (127.21.84.133:42509)
I20240501 08:10:53.503463 23098 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.133:49585
I20240501 08:10:53.506753 21842 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 08:10:53.514086 23545 heartbeater.cc:503] Master 127.21.84.190:38497 was elected leader, sending a full tablet report...
I20240501 08:10:53.522426 23098 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55552:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 08:10:53.558883 23219 tablet_service.cc:1467] Processing CreateTablet for tablet 4ce41edba0b74dbc9064b13ba1545eb9 (DEFAULT_TABLE table=TestTable [id=e7ca5ea859d74fabb0cd73febb72002d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:53.558883 23611 tablet_service.cc:1467] Processing CreateTablet for tablet 4ce41edba0b74dbc9064b13ba1545eb9 (DEFAULT_TABLE table=TestTable [id=e7ca5ea859d74fabb0cd73febb72002d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:53.559224 23480 tablet_service.cc:1467] Processing CreateTablet for tablet 4ce41edba0b74dbc9064b13ba1545eb9 (DEFAULT_TABLE table=TestTable [id=e7ca5ea859d74fabb0cd73febb72002d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:53.560436 23219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ce41edba0b74dbc9064b13ba1545eb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:53.560457 23611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ce41edba0b74dbc9064b13ba1545eb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:53.560715 23480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ce41edba0b74dbc9064b13ba1545eb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:53.571382 23825 tablet_bootstrap.cc:492] T 4ce41edba0b74dbc9064b13ba1545eb9 P 9c47bb4ff8cc4b2b9311b9810f9c584f: Bootstrap starting.
I20240501 08:10:53.573819 23827 tablet_bootstrap.cc:492] T 4ce41edba0b74dbc9064b13ba1545eb9 P b17af124243a4af3bf8a430803a498e0: Bootstrap starting.
I20240501 08:10:53.574687 23826 tablet_bootstrap.cc:492] T 4ce41edba0b74dbc9064b13ba1545eb9 P 06d814f013b240af85b4209cb43b0cd7: Bootstrap starting.
I20240501 08:10:53.575645 23825 tablet_bootstrap.cc:654] T 4ce41edba0b74dbc9064b13ba1545eb9 P 9c47bb4ff8cc4b2b9311b9810f9c584f: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:53.576984 23825 log.cc:824] T 4ce41edba0b74dbc9064b13ba1545eb9 P 9c47bb4ff8cc4b2b9311b9810f9c584f: Log is configured to *not* fsync() on all Append() calls
I20240501 08:10:53.578433 23826 tablet_bootstrap.cc:654] T 4ce41edba0b74dbc9064b13ba1545eb9 P 06d814f013b240af85b4209cb43b0cd7: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:53.579833 23826 log.cc:824] T 4ce41edba0b74dbc9064b13ba1545eb9 P 06d814f013b240af85b4209cb43b0cd7: Log is configured to *not* fsync() on all Append() calls
I20240501 08:10:53.581249 23827 tablet_bootstrap.cc:654] T 4ce41edba0b74dbc9064b13ba1545eb9 P b17af124243a4af3bf8a430803a498e0: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:53.582556 23825 tablet_bootstrap.cc:492] T 4ce41edba0b74dbc9064b13ba1545eb9 P 9c47bb4ff8cc4b2b9311b9810f9c584f: No bootstrap required, opened a new log
I20240501 08:10:53.582583 23827 log.cc:824] T 4ce41edba0b74dbc9064b13ba1545eb9 P b17af124243a4af3bf8a430803a498e0: Log is configured to *not* fsync() on all Append() calls
I20240501 08:10:53.582772 23826 tablet_bootstrap.cc:492] T 4ce41edba0b74dbc9064b13ba1545eb9 P 06d814f013b240af85b4209cb43b0cd7: No bootstrap required, opened a new log
I20240501 08:10:53.582904 23825 ts_tablet_manager.cc:1367] T 4ce41edba0b74dbc9064b13ba1545eb9 P 9c47bb4ff8cc4b2b9311b9810f9c584f: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20240501 08:10:53.583055 23826 ts_tablet_manager.cc:1367] T 4ce41edba0b74dbc9064b13ba1545eb9 P 06d814f013b240af85b4209cb43b0cd7: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240501 08:10:53.590517 23826 raft_consensus.cc:348] T 4ce41edba0b74dbc9064b13ba1545eb9 P 06d814f013b240af85b4209cb43b0cd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d814f013b240af85b4209cb43b0cd7" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 39167 } } peers { permanent_uuid: "b17af124243a4af3bf8a430803a498e0" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 35337 } } peers { permanent_uuid: "9c47bb4ff8cc4b2b9311b9810f9c584f" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 45969 } }
I20240501 08:10:53.590953 23826 raft_consensus.cc:374] T 4ce41edba0b74dbc9064b13ba1545eb9 P 06d814f013b240af85b4209cb43b0cd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:53.591071 23826 raft_consensus.cc:729] T 4ce41edba0b74dbc9064b13ba1545eb9 P 06d814f013b240af85b4209cb43b0cd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06d814f013b240af85b4209cb43b0cd7, State: Initialized, Role: FOLLOWER
I20240501 08:10:53.591758 23826 consensus_queue.cc:260] T 4ce41edba0b74dbc9064b13ba1545eb9 P 06d814f013b240af85b4209cb43b0cd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d814f013b240af85b4209cb43b0cd7" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 39167 } } peers { permanent_uuid: "b17af124243a4af3bf8a430803a498e0" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 35337 } } peers { permanent_uuid: "9c47bb4ff8cc4b2b9311b9810f9c584f" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 45969 } }
I20240501 08:10:53.592717 23827 tablet_bootstrap.cc:492] T 4ce41edba0b74dbc9064b13ba1545eb9 P b17af124243a4af3bf8a430803a498e0: No bootstrap required, opened a new log
I20240501 08:10:53.593103 23827 ts_tablet_manager.cc:1367] T 4ce41edba0b74dbc9064b13ba1545eb9 P b17af124243a4af3bf8a430803a498e0: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.008s
I20240501 08:10:53.593461 23826 ts_tablet_manager.cc:1398] T 4ce41edba0b74dbc9064b13ba1545eb9 P 06d814f013b240af85b4209cb43b0cd7: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20240501 08:10:53.594605 23825 raft_consensus.cc:348] T 4ce41edba0b74dbc9064b13ba1545eb9 P 9c47bb4ff8cc4b2b9311b9810f9c584f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d814f013b240af85b4209cb43b0cd7" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 39167 } } peers { permanent_uuid: "b17af124243a4af3bf8a430803a498e0" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 35337 } } peers { permanent_uuid: "9c47bb4ff8cc4b2b9311b9810f9c584f" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 45969 } }
I20240501 08:10:53.594974 23825 raft_consensus.cc:374] T 4ce41edba0b74dbc9064b13ba1545eb9 P 9c47bb4ff8cc4b2b9311b9810f9c584f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:53.595064 23825 raft_consensus.cc:729] T 4ce41edba0b74dbc9064b13ba1545eb9 P 9c47bb4ff8cc4b2b9311b9810f9c584f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c47bb4ff8cc4b2b9311b9810f9c584f, State: Initialized, Role: FOLLOWER
I20240501 08:10:53.595728 23825 consensus_queue.cc:260] T 4ce41edba0b74dbc9064b13ba1545eb9 P 9c47bb4ff8cc4b2b9311b9810f9c584f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d814f013b240af85b4209cb43b0cd7" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 39167 } } peers { permanent_uuid: "b17af124243a4af3bf8a430803a498e0" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 35337 } } peers { permanent_uuid: "9c47bb4ff8cc4b2b9311b9810f9c584f" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 45969 } }
I20240501 08:10:53.597290 23825 ts_tablet_manager.cc:1398] T 4ce41edba0b74dbc9064b13ba1545eb9 P 9c47bb4ff8cc4b2b9311b9810f9c584f: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20240501 08:10:53.601356 23827 raft_consensus.cc:348] T 4ce41edba0b74dbc9064b13ba1545eb9 P b17af124243a4af3bf8a430803a498e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d814f013b240af85b4209cb43b0cd7" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 39167 } } peers { permanent_uuid: "b17af124243a4af3bf8a430803a498e0" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 35337 } } peers { permanent_uuid: "9c47bb4ff8cc4b2b9311b9810f9c584f" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 45969 } }
I20240501 08:10:53.601706 23827 raft_consensus.cc:374] T 4ce41edba0b74dbc9064b13ba1545eb9 P b17af124243a4af3bf8a430803a498e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:53.601830 23827 raft_consensus.cc:729] T 4ce41edba0b74dbc9064b13ba1545eb9 P b17af124243a4af3bf8a430803a498e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b17af124243a4af3bf8a430803a498e0, State: Initialized, Role: FOLLOWER
I20240501 08:10:53.602408 23827 consensus_queue.cc:260] T 4ce41edba0b74dbc9064b13ba1545eb9 P b17af124243a4af3bf8a430803a498e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d814f013b240af85b4209cb43b0cd7" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 39167 } } peers { permanent_uuid: "b17af124243a4af3bf8a430803a498e0" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 35337 } } peers { permanent_uuid: "9c47bb4ff8cc4b2b9311b9810f9c584f" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 45969 } }
I20240501 08:10:53.603322 23676 heartbeater.cc:503] Master 127.21.84.190:38497 was elected leader, sending a full tablet report...
I20240501 08:10:53.603904 23827 ts_tablet_manager.cc:1398] T 4ce41edba0b74dbc9064b13ba1545eb9 P b17af124243a4af3bf8a430803a498e0: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
W20240501 08:10:53.746203 23546 tablet.cc:2369] T 4ce41edba0b74dbc9064b13ba1545eb9 P 9c47bb4ff8cc4b2b9311b9810f9c584f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 08:10:53.776029 23677 tablet.cc:2369] T 4ce41edba0b74dbc9064b13ba1545eb9 P b17af124243a4af3bf8a430803a498e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 08:10:53.832090 23285 tablet.cc:2369] T 4ce41edba0b74dbc9064b13ba1545eb9 P 06d814f013b240af85b4209cb43b0cd7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:10:53.881531 23832 raft_consensus.cc:482] T 4ce41edba0b74dbc9064b13ba1545eb9 P 9c47bb4ff8cc4b2b9311b9810f9c584f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:53.881841 23832 raft_consensus.cc:504] T 4ce41edba0b74dbc9064b13ba1545eb9 P 9c47bb4ff8cc4b2b9311b9810f9c584f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d814f013b240af85b4209cb43b0cd7" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 39167 } } peers { permanent_uuid: "b17af124243a4af3bf8a430803a498e0" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 35337 } } peers { permanent_uuid: "9c47bb4ff8cc4b2b9311b9810f9c584f" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 45969 } }
I20240501 08:10:53.883761 23832 leader_election.cc:288] T 4ce41edba0b74dbc9064b13ba1545eb9 P 9c47bb4ff8cc4b2b9311b9810f9c584f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06d814f013b240af85b4209cb43b0cd7 (127.21.84.129:39167), b17af124243a4af3bf8a430803a498e0 (127.21.84.132:35337)
I20240501 08:10:53.892094 23239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ce41edba0b74dbc9064b13ba1545eb9" candidate_uuid: "9c47bb4ff8cc4b2b9311b9810f9c584f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06d814f013b240af85b4209cb43b0cd7" is_pre_election: true
I20240501 08:10:53.892537 23239 raft_consensus.cc:2454] T 4ce41edba0b74dbc9064b13ba1545eb9 P 06d814f013b240af85b4209cb43b0cd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c47bb4ff8cc4b2b9311b9810f9c584f in term 0.
I20240501 08:10:53.893452 23433 leader_election.cc:302] T 4ce41edba0b74dbc9064b13ba1545eb9 P 9c47bb4ff8cc4b2b9311b9810f9c584f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06d814f013b240af85b4209cb43b0cd7, 9c47bb4ff8cc4b2b9311b9810f9c584f; no voters:
I20240501 08:10:53.893592 23631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ce41edba0b74dbc9064b13ba1545eb9" candidate_uuid: "9c47bb4ff8cc4b2b9311b9810f9c584f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b17af124243a4af3bf8a430803a498e0" is_pre_election: true
I20240501 08:10:53.893883 23832 raft_consensus.cc:2781] T 4ce41edba0b74dbc9064b13ba1545eb9 P 9c47bb4ff8cc4b2b9311b9810f9c584f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:53.894061 23832 raft_consensus.cc:482] T 4ce41edba0b74dbc9064b13ba1545eb9 P 9c47bb4ff8cc4b2b9311b9810f9c584f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:53.894161 23832 raft_consensus.cc:3037] T 4ce41edba0b74dbc9064b13ba1545eb9 P 9c47bb4ff8cc4b2b9311b9810f9c584f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:53.894207 23631 raft_consensus.cc:2454] T 4ce41edba0b74dbc9064b13ba1545eb9 P b17af124243a4af3bf8a430803a498e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c47bb4ff8cc4b2b9311b9810f9c584f in term 0.
I20240501 08:10:53.964390 23832 raft_consensus.cc:504] T 4ce41edba0b74dbc9064b13ba1545eb9 P 9c47bb4ff8cc4b2b9311b9810f9c584f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d814f013b240af85b4209cb43b0cd7" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 39167 } } peers { permanent_uuid: "b17af124243a4af3bf8a430803a498e0" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 35337 } } peers { permanent_uuid: "9c47bb4ff8cc4b2b9311b9810f9c584f" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 45969 } }
I20240501 08:10:53.965164 23832 leader_election.cc:288] T 4ce41edba0b74dbc9064b13ba1545eb9 P 9c47bb4ff8cc4b2b9311b9810f9c584f [CANDIDATE]: Term 1 election: Requested vote from peers 06d814f013b240af85b4209cb43b0cd7 (127.21.84.129:39167), b17af124243a4af3bf8a430803a498e0 (127.21.84.132:35337)
I20240501 08:10:53.965802 23239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ce41edba0b74dbc9064b13ba1545eb9" candidate_uuid: "9c47bb4ff8cc4b2b9311b9810f9c584f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06d814f013b240af85b4209cb43b0cd7"
I20240501 08:10:53.965894 23631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ce41edba0b74dbc9064b13ba1545eb9" candidate_uuid: "9c47bb4ff8cc4b2b9311b9810f9c584f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b17af124243a4af3bf8a430803a498e0"
I20240501 08:10:53.966071 23239 raft_consensus.cc:3037] T 4ce41edba0b74dbc9064b13ba1545eb9 P 06d814f013b240af85b4209cb43b0cd7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:53.966115 23631 raft_consensus.cc:3037] T 4ce41edba0b74dbc9064b13ba1545eb9 P b17af124243a4af3bf8a430803a498e0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:54.065774 23631 raft_consensus.cc:2454] T 4ce41edba0b74dbc9064b13ba1545eb9 P b17af124243a4af3bf8a430803a498e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c47bb4ff8cc4b2b9311b9810f9c584f in term 1.
I20240501 08:10:54.065774 23239 raft_consensus.cc:2454] T 4ce41edba0b74dbc9064b13ba1545eb9 P 06d814f013b240af85b4209cb43b0cd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c47bb4ff8cc4b2b9311b9810f9c584f in term 1.
I20240501 08:10:54.066438 23435 leader_election.cc:302] T 4ce41edba0b74dbc9064b13ba1545eb9 P 9c47bb4ff8cc4b2b9311b9810f9c584f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c47bb4ff8cc4b2b9311b9810f9c584f, b17af124243a4af3bf8a430803a498e0; no voters:
I20240501 08:10:54.066867 23832 raft_consensus.cc:2781] T 4ce41edba0b74dbc9064b13ba1545eb9 P 9c47bb4ff8cc4b2b9311b9810f9c584f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:54.067369 23832 raft_consensus.cc:686] T 4ce41edba0b74dbc9064b13ba1545eb9 P 9c47bb4ff8cc4b2b9311b9810f9c584f [term 1 LEADER]: Becoming Leader. State: Replica: 9c47bb4ff8cc4b2b9311b9810f9c584f, State: Running, Role: LEADER
I20240501 08:10:54.067865 23832 consensus_queue.cc:237] T 4ce41edba0b74dbc9064b13ba1545eb9 P 9c47bb4ff8cc4b2b9311b9810f9c584f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d814f013b240af85b4209cb43b0cd7" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 39167 } } peers { permanent_uuid: "b17af124243a4af3bf8a430803a498e0" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 35337 } } peers { permanent_uuid: "9c47bb4ff8cc4b2b9311b9810f9c584f" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 45969 } }
I20240501 08:10:54.072844 23098 catalog_manager.cc:5484] T 4ce41edba0b74dbc9064b13ba1545eb9 P 9c47bb4ff8cc4b2b9311b9810f9c584f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c47bb4ff8cc4b2b9311b9810f9c584f (127.21.84.131). New cstate: current_term: 1 leader_uuid: "9c47bb4ff8cc4b2b9311b9810f9c584f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06d814f013b240af85b4209cb43b0cd7" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 39167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b17af124243a4af3bf8a430803a498e0" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 35337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c47bb4ff8cc4b2b9311b9810f9c584f" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 45969 } health_report { overall_health: HEALTHY } } }
I20240501 08:10:54.115305 21842 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 08:10:54.117409 21842 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 06d814f013b240af85b4209cb43b0cd7 to finish bootstrapping
I20240501 08:10:54.125373 21842 ts_itest-base.cc:233] Waiting for 0 tablets on tserver acc1d95756c441879c7faa6760d0dfc9 to finish bootstrapping
I20240501 08:10:54.134953 21842 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 9c47bb4ff8cc4b2b9311b9810f9c584f to finish bootstrapping
I20240501 08:10:54.142827 21842 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b17af124243a4af3bf8a430803a498e0 to finish bootstrapping
I20240501 08:10:54.150219 21842 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 248cfd6a113e436abf83f58c7800ddf5 to finish bootstrapping
I20240501 08:10:54.159242 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 23157
I20240501 08:10:54.160218 23279 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:10:54.249214 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 23157
I20240501 08:10:54.505892 23807 heartbeater.cc:503] Master 127.21.84.190:38497 was elected leader, sending a full tablet report...
W20240501 08:10:54.615238 23433 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.84.129:39167: connect: Connection refused (error 111)
W20240501 08:10:54.616693 23433 consensus_peers.cc:485] T 4ce41edba0b74dbc9064b13ba1545eb9 P 9c47bb4ff8cc4b2b9311b9810f9c584f -> Peer 06d814f013b240af85b4209cb43b0cd7 (127.21.84.129:39167): Couldn't send request to peer 06d814f013b240af85b4209cb43b0cd7. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:39167: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:10:54.650615 23850 consensus_queue.cc:1035] T 4ce41edba0b74dbc9064b13ba1545eb9 P 9c47bb4ff8cc4b2b9311b9810f9c584f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b17af124243a4af3bf8a430803a498e0" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 35337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:54.756425 23742 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:54.759052 23611 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:54.760437 23480 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:10:54.762820 23350 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:54.891518 23742 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:54.892838 23480 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:10:54.902849 23350 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:54.906026 23611 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:55.087881 23742 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:55.088584 23350 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:55.089656 23480 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:10:55.092006 23611 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:55.730227 23742 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:55.737254 23350 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:55.738025 23611 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:55.739889 23480 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:10:55.883191 23480 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:10:55.887157 23742 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:55.891573 23350 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:55.893855 23611 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:56.048813 23742 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:56.059782 23480 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:10:56.063141 23350 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:56.065138 23611 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:56.205585 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 23288
I20240501 08:10:56.207018 23410 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:10:56.297861 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 23288
I20240501 08:10:56.315361 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 23418
I20240501 08:10:56.316443 23540 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:10:56.414139 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 23418
I20240501 08:10:56.432293 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 23548
I20240501 08:10:56.433332 23671 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:10:56.536895 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 23548
I20240501 08:10:56.554841 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 23680
I20240501 08:10:56.555891 23802 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:10:56.648394 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 23680
I20240501 08:10:56.665716 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 23067
I20240501 08:10:56.666839 23128 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:10:56.783383 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 23067
2024-05-01T08:10:56Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/1 (6349 ms)
[----------] 1 test from RebalanceStartCriteriaTest (6349 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/1
I20240501 08:10:56.827318 21842 test_util.cc:268] Using random seed: 112264094
I20240501 08:10:56.829249 21842 ts_itest-base.cc:112] Starting cluster with:
I20240501 08:10:56.829319 21842 ts_itest-base.cc:113] --------------
I20240501 08:10:56.829372 21842 ts_itest-base.cc:114] 7 tablet servers
I20240501 08:10:56.829424 21842 ts_itest-base.cc:115] 1 replicas per TS
I20240501 08:10:56.829505 21842 ts_itest-base.cc:116] --------------
2024-05-01T08:10:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T08:10:56Z Disabled control of system clock
I20240501 08:10:56.854099 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.84.190:43065
--webserver_interface=127.21.84.190
--webserver_port=0
--builtin_ntp_servers=127.21.84.148:38087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.84.190:43065
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240501 08:10:57.191069 24056 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 08:10:57.191432 24056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:57.191546 24056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:57.191654 24056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:57.199342 24056 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:10:57.199512 24056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 08:10:57.199760 24056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:57.199862 24056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 08:10:57.199960 24056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 08:10:57.209522 24056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:38087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.21.84.190:43065
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.84.190:43065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.21.84.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:57.210947 24056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:57.212694 24056 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:57.224108 24061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:57.224984 24064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:57.225732 24062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:57.226637 24056 server_base.cc:1034] running on GCE node
I20240501 08:10:57.227504 24056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:57.229041 24056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:57.230274 24056 hybrid_clock.cc:648] HybridClock initialized: now 1714551057230187 us; error 54 us; skew 500 ppm
I20240501 08:10:57.230670 24056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:57.233281 24056 webserver.cc:460] Webserver started at http://127.21.84.190:46869/ using document root <none> and password file <none>
I20240501 08:10:57.234097 24056 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:57.234230 24056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:57.234541 24056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:57.237358 24056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "1ba60b54c62540a8a64e0fcd6b4eaa8f"
format_stamp: "Formatted at 2024-05-01 08:10:57 on dist-test-slave-9cgk"
I20240501 08:10:57.238261 24056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "1ba60b54c62540a8a64e0fcd6b4eaa8f"
format_stamp: "Formatted at 2024-05-01 08:10:57 on dist-test-slave-9cgk"
I20240501 08:10:57.243940 24056 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 08:10:57.247460 24070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:57.248984 24056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 08:10:57.249198 24056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/wal
uuid: "1ba60b54c62540a8a64e0fcd6b4eaa8f"
format_stamp: "Formatted at 2024-05-01 08:10:57 on dist-test-slave-9cgk"
I20240501 08:10:57.249401 24056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:57.267385 24056 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:57.268626 24056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:57.269033 24056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:57.288113 24056 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.190:43065
I20240501 08:10:57.288133 24121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.190:43065 every 8 connection(s)
I20240501 08:10:57.289646 24056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 08:10:57.292001 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 24056
I20240501 08:10:57.292188 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 08:10:57.293870 24122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:57.306602 24122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba60b54c62540a8a64e0fcd6b4eaa8f: Bootstrap starting.
I20240501 08:10:57.309638 24122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ba60b54c62540a8a64e0fcd6b4eaa8f: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:57.310722 24122 log.cc:824] T 00000000000000000000000000000000 P 1ba60b54c62540a8a64e0fcd6b4eaa8f: Log is configured to *not* fsync() on all Append() calls
I20240501 08:10:57.313334 24122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba60b54c62540a8a64e0fcd6b4eaa8f: No bootstrap required, opened a new log
I20240501 08:10:57.319919 24122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ba60b54c62540a8a64e0fcd6b4eaa8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba60b54c62540a8a64e0fcd6b4eaa8f" member_type: VOTER last_known_addr { host: "127.21.84.190" port: 43065 } }
I20240501 08:10:57.320319 24122 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ba60b54c62540a8a64e0fcd6b4eaa8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:57.320425 24122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ba60b54c62540a8a64e0fcd6b4eaa8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ba60b54c62540a8a64e0fcd6b4eaa8f, State: Initialized, Role: FOLLOWER
I20240501 08:10:57.321082 24122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ba60b54c62540a8a64e0fcd6b4eaa8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba60b54c62540a8a64e0fcd6b4eaa8f" member_type: VOTER last_known_addr { host: "127.21.84.190" port: 43065 } }
I20240501 08:10:57.321317 24122 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ba60b54c62540a8a64e0fcd6b4eaa8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:10:57.321441 24122 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ba60b54c62540a8a64e0fcd6b4eaa8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:10:57.321605 24122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ba60b54c62540a8a64e0fcd6b4eaa8f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:57.324151 24122 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ba60b54c62540a8a64e0fcd6b4eaa8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba60b54c62540a8a64e0fcd6b4eaa8f" member_type: VOTER last_known_addr { host: "127.21.84.190" port: 43065 } }
I20240501 08:10:57.324666 24122 leader_election.cc:302] T 00000000000000000000000000000000 P 1ba60b54c62540a8a64e0fcd6b4eaa8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ba60b54c62540a8a64e0fcd6b4eaa8f; no voters:
I20240501 08:10:57.325271 24122 leader_election.cc:288] T 00000000000000000000000000000000 P 1ba60b54c62540a8a64e0fcd6b4eaa8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:10:57.325407 24127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ba60b54c62540a8a64e0fcd6b4eaa8f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:57.326047 24127 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ba60b54c62540a8a64e0fcd6b4eaa8f [term 1 LEADER]: Becoming Leader. State: Replica: 1ba60b54c62540a8a64e0fcd6b4eaa8f, State: Running, Role: LEADER
I20240501 08:10:57.326654 24127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ba60b54c62540a8a64e0fcd6b4eaa8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba60b54c62540a8a64e0fcd6b4eaa8f" member_type: VOTER last_known_addr { host: "127.21.84.190" port: 43065 } }
I20240501 08:10:57.327368 24122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ba60b54c62540a8a64e0fcd6b4eaa8f [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:10:57.329423 24129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba60b54c62540a8a64e0fcd6b4eaa8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ba60b54c62540a8a64e0fcd6b4eaa8f. Latest consensus state: current_term: 1 leader_uuid: "1ba60b54c62540a8a64e0fcd6b4eaa8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba60b54c62540a8a64e0fcd6b4eaa8f" member_type: VOTER last_known_addr { host: "127.21.84.190" port: 43065 } } }
I20240501 08:10:57.329402 24128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba60b54c62540a8a64e0fcd6b4eaa8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ba60b54c62540a8a64e0fcd6b4eaa8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba60b54c62540a8a64e0fcd6b4eaa8f" member_type: VOTER last_known_addr { host: "127.21.84.190" port: 43065 } } }
I20240501 08:10:57.329798 24129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba60b54c62540a8a64e0fcd6b4eaa8f [sys.catalog]: This master's current role is: LEADER
I20240501 08:10:57.329795 24128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba60b54c62540a8a64e0fcd6b4eaa8f [sys.catalog]: This master's current role is: LEADER
I20240501 08:10:57.331703 24132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:10:57.337188 24132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:10:57.346778 24132 catalog_manager.cc:1347] Generated new cluster ID: 365a589f87474c2ba9dff97d4616c110
I20240501 08:10:57.346916 24132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:10:57.362411 24132 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:10:57.364034 24132 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:10:57.382107 24132 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1ba60b54c62540a8a64e0fcd6b4eaa8f: Generated new TSK 0
I20240501 08:10:57.383016 24132 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:10:57.390843 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.129:0
--local_ip_for_outbound_sockets=127.21.84.129
--webserver_interface=127.21.84.129
--webserver_port=0
--tserver_master_addrs=127.21.84.190:43065
--builtin_ntp_servers=127.21.84.148:38087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 08:10:57.725143 24146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:57.725469 24146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:57.725575 24146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:57.733538 24146 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:10:57.733670 24146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:57.733834 24146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.129
I20240501 08:10:57.743255 24146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:38087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.21.84.129
--webserver_port=0
--tserver_master_addrs=127.21.84.190:43065
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.129
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:57.744727 24146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:57.746507 24146 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:57.759125 24152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:57.759958 24151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:57.760660 24154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:57.761540 24146 server_base.cc:1034] running on GCE node
I20240501 08:10:57.762217 24146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:57.763180 24146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:57.764389 24146 hybrid_clock.cc:648] HybridClock initialized: now 1714551057764334 us; error 45 us; skew 500 ppm
I20240501 08:10:57.764724 24146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:57.767015 24146 webserver.cc:460] Webserver started at http://127.21.84.129:42775/ using document root <none> and password file <none>
I20240501 08:10:57.767784 24146 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:57.767897 24146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:57.768199 24146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:57.770896 24146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "fb59098726224e7bae383f5c11820b39"
format_stamp: "Formatted at 2024-05-01 08:10:57 on dist-test-slave-9cgk"
I20240501 08:10:57.771673 24146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "fb59098726224e7bae383f5c11820b39"
format_stamp: "Formatted at 2024-05-01 08:10:57 on dist-test-slave-9cgk"
I20240501 08:10:57.777012 24146 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 08:10:57.780468 24160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:57.781849 24146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240501 08:10:57.782052 24146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "fb59098726224e7bae383f5c11820b39"
format_stamp: "Formatted at 2024-05-01 08:10:57 on dist-test-slave-9cgk"
I20240501 08:10:57.782245 24146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:57.796204 24146 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:57.797049 24146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:57.797322 24146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:57.798328 24146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:57.799896 24146 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:57.799983 24146 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:57.800093 24146 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:57.800164 24146 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:57.832015 24146 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.129:44759
I20240501 08:10:57.832070 24272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.129:44759 every 8 connection(s)
I20240501 08:10:57.833415 24146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 08:10:57.840322 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 24146
I20240501 08:10:57.840553 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 08:10:57.843830 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.130:0
--local_ip_for_outbound_sockets=127.21.84.130
--webserver_interface=127.21.84.130
--webserver_port=0
--tserver_master_addrs=127.21.84.190:43065
--builtin_ntp_servers=127.21.84.148:38087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:10:57.846959 24273 heartbeater.cc:347] Connected to a master server at 127.21.84.190:43065
I20240501 08:10:57.847263 24273 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:57.848106 24273 heartbeater.cc:511] Master 127.21.84.190:43065 requested a full tablet report, sending...
I20240501 08:10:57.850407 24087 ts_manager.cc:194] Registered new tserver with Master: fb59098726224e7bae383f5c11820b39 (127.21.84.129:44759)
I20240501 08:10:57.851934 24087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.129:47161
W20240501 08:10:58.175079 24277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:58.175400 24277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:58.175525 24277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:58.183563 24277 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:10:58.183701 24277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:58.184010 24277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.130
I20240501 08:10:58.193428 24277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:38087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.21.84.130
--webserver_port=0
--tserver_master_addrs=127.21.84.190:43065
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.130
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:58.194900 24277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:58.196682 24277 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:58.209805 24282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:58.210436 24283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:58.211405 24285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:58.212240 24277 server_base.cc:1034] running on GCE node
I20240501 08:10:58.212935 24277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:58.213954 24277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:58.215183 24277 hybrid_clock.cc:648] HybridClock initialized: now 1714551058215123 us; error 59 us; skew 500 ppm
I20240501 08:10:58.215525 24277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:58.223233 24277 webserver.cc:460] Webserver started at http://127.21.84.130:35079/ using document root <none> and password file <none>
I20240501 08:10:58.224109 24277 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:58.224243 24277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:58.224589 24277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:58.227499 24277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "213ccbb05e6f44deb4a34d0a7041f7de"
format_stamp: "Formatted at 2024-05-01 08:10:58 on dist-test-slave-9cgk"
I20240501 08:10:58.228345 24277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "213ccbb05e6f44deb4a34d0a7041f7de"
format_stamp: "Formatted at 2024-05-01 08:10:58 on dist-test-slave-9cgk"
I20240501 08:10:58.234016 24277 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240501 08:10:58.237509 24291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:58.238994 24277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:10:58.239168 24277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "213ccbb05e6f44deb4a34d0a7041f7de"
format_stamp: "Formatted at 2024-05-01 08:10:58 on dist-test-slave-9cgk"
I20240501 08:10:58.239370 24277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:58.261768 24277 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:58.262710 24277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:58.262966 24277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:58.263906 24277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:58.265400 24277 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:58.265506 24277 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:58.265592 24277 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:58.265643 24277 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:58.297751 24277 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.130:37857
I20240501 08:10:58.297770 24403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.130:37857 every 8 connection(s)
I20240501 08:10:58.299501 24277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 08:10:58.303155 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 24277
I20240501 08:10:58.303319 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 08:10:58.306922 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.131:0
--local_ip_for_outbound_sockets=127.21.84.131
--webserver_interface=127.21.84.131
--webserver_port=0
--tserver_master_addrs=127.21.84.190:43065
--builtin_ntp_servers=127.21.84.148:38087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:10:58.313680 24404 heartbeater.cc:347] Connected to a master server at 127.21.84.190:43065
I20240501 08:10:58.314057 24404 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:58.315112 24404 heartbeater.cc:511] Master 127.21.84.190:43065 requested a full tablet report, sending...
I20240501 08:10:58.316780 24087 ts_manager.cc:194] Registered new tserver with Master: 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130:37857)
I20240501 08:10:58.317551 24087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.130:43567
W20240501 08:10:58.644641 24407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:58.644960 24407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:58.645085 24407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:58.652737 24407 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:10:58.652875 24407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:58.652990 24407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.131
I20240501 08:10:58.662325 24407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:38087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.21.84.131
--webserver_port=0
--tserver_master_addrs=127.21.84.190:43065
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.131
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:58.663684 24407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:58.665410 24407 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:58.678499 24413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:58.679134 24412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:58.679819 24415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:58.680817 24407 server_base.cc:1034] running on GCE node
I20240501 08:10:58.681529 24407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:58.682626 24407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:58.683844 24407 hybrid_clock.cc:648] HybridClock initialized: now 1714551058683796 us; error 45 us; skew 500 ppm
I20240501 08:10:58.684167 24407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:58.686589 24407 webserver.cc:460] Webserver started at http://127.21.84.131:38437/ using document root <none> and password file <none>
I20240501 08:10:58.687372 24407 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:58.687494 24407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:58.687801 24407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:58.690558 24407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b"
format_stamp: "Formatted at 2024-05-01 08:10:58 on dist-test-slave-9cgk"
I20240501 08:10:58.691330 24407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b"
format_stamp: "Formatted at 2024-05-01 08:10:58 on dist-test-slave-9cgk"
I20240501 08:10:58.696548 24407 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 08:10:58.700074 24421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:58.701503 24407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20240501 08:10:58.701665 24407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b"
format_stamp: "Formatted at 2024-05-01 08:10:58 on dist-test-slave-9cgk"
I20240501 08:10:58.701871 24407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:58.721529 24407 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:58.722465 24407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:58.722741 24407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:58.723707 24407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:58.725167 24407 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:58.725260 24407 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:58.725365 24407 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:58.725431 24407 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:58.756743 24407 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.131:35031
I20240501 08:10:58.756794 24533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.131:35031 every 8 connection(s)
I20240501 08:10:58.758270 24407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 08:10:58.767894 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 24407
I20240501 08:10:58.768092 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 08:10:58.771430 24534 heartbeater.cc:347] Connected to a master server at 127.21.84.190:43065
I20240501 08:10:58.771457 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.132:0
--local_ip_for_outbound_sockets=127.21.84.132
--webserver_interface=127.21.84.132
--webserver_port=0
--tserver_master_addrs=127.21.84.190:43065
--builtin_ntp_servers=127.21.84.148:38087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:10:58.771776 24534 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:58.772614 24534 heartbeater.cc:511] Master 127.21.84.190:43065 requested a full tablet report, sending...
I20240501 08:10:58.774139 24087 ts_manager.cc:194] Registered new tserver with Master: f0c82b0e87d84b5eb3b2fd4f3e63ff1b (127.21.84.131:35031)
I20240501 08:10:58.775300 24087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.131:46701
I20240501 08:10:58.855583 24273 heartbeater.cc:503] Master 127.21.84.190:43065 was elected leader, sending a full tablet report...
W20240501 08:10:59.106688 24537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:59.106998 24537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:59.107097 24537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:59.114499 24537 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:10:59.114634 24537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:59.114749 24537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.132
I20240501 08:10:59.126894 24537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:38087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.21.84.132
--webserver_port=0
--tserver_master_addrs=127.21.84.190:43065
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.132
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:59.128310 24537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:59.130126 24537 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:59.142560 24543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:59.143402 24542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:59.144066 24545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:59.145028 24537 server_base.cc:1034] running on GCE node
I20240501 08:10:59.145629 24537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:59.146670 24537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:59.147881 24537 hybrid_clock.cc:648] HybridClock initialized: now 1714551059147818 us; error 55 us; skew 500 ppm
I20240501 08:10:59.148231 24537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:59.150712 24537 webserver.cc:460] Webserver started at http://127.21.84.132:40905/ using document root <none> and password file <none>
I20240501 08:10:59.151463 24537 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:59.151571 24537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:59.151944 24537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:59.155233 24537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "5ce52428c87b46369add4a8afaaef67c"
format_stamp: "Formatted at 2024-05-01 08:10:59 on dist-test-slave-9cgk"
I20240501 08:10:59.156050 24537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "5ce52428c87b46369add4a8afaaef67c"
format_stamp: "Formatted at 2024-05-01 08:10:59 on dist-test-slave-9cgk"
I20240501 08:10:59.161511 24537 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240501 08:10:59.165201 24551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:59.166662 24537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:10:59.166833 24537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "5ce52428c87b46369add4a8afaaef67c"
format_stamp: "Formatted at 2024-05-01 08:10:59 on dist-test-slave-9cgk"
I20240501 08:10:59.167037 24537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:59.198992 24537 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:59.200001 24537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:59.200289 24537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:59.201328 24537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:59.202937 24537 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:59.203037 24537 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:59.203155 24537 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:59.203245 24537 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:59.235666 24537 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.132:43389
I20240501 08:10:59.235724 24663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.132:43389 every 8 connection(s)
I20240501 08:10:59.237005 24537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 08:10:59.240734 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 24537
I20240501 08:10:59.240911 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 08:10:59.244345 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.133:0
--local_ip_for_outbound_sockets=127.21.84.133
--webserver_interface=127.21.84.133
--webserver_port=0
--tserver_master_addrs=127.21.84.190:43065
--builtin_ntp_servers=127.21.84.148:38087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:10:59.251471 24664 heartbeater.cc:347] Connected to a master server at 127.21.84.190:43065
I20240501 08:10:59.251806 24664 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:59.252665 24664 heartbeater.cc:511] Master 127.21.84.190:43065 requested a full tablet report, sending...
I20240501 08:10:59.254321 24087 ts_manager.cc:194] Registered new tserver with Master: 5ce52428c87b46369add4a8afaaef67c (127.21.84.132:43389)
I20240501 08:10:59.255570 24087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.132:52501
I20240501 08:10:59.321118 24404 heartbeater.cc:503] Master 127.21.84.190:43065 was elected leader, sending a full tablet report...
W20240501 08:10:59.584085 24667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:59.584414 24667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:59.584502 24667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:59.592281 24667 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:10:59.592417 24667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:59.592535 24667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.133
I20240501 08:10:59.601600 24667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:38087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.21.84.133
--webserver_port=0
--tserver_master_addrs=127.21.84.190:43065
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.133
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:59.603021 24667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:59.604774 24667 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:59.617605 24673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:59.618199 24672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:59.618803 24675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:59.619868 24667 server_base.cc:1034] running on GCE node
I20240501 08:10:59.620507 24667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:59.621513 24667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:59.622713 24667 hybrid_clock.cc:648] HybridClock initialized: now 1714551059622666 us; error 45 us; skew 500 ppm
I20240501 08:10:59.623047 24667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:59.625272 24667 webserver.cc:460] Webserver started at http://127.21.84.133:32863/ using document root <none> and password file <none>
I20240501 08:10:59.626087 24667 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:59.626204 24667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:59.626506 24667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:59.629277 24667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "962160cac57040678e5cacd4b8b73c4b"
format_stamp: "Formatted at 2024-05-01 08:10:59 on dist-test-slave-9cgk"
I20240501 08:10:59.630106 24667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "962160cac57040678e5cacd4b8b73c4b"
format_stamp: "Formatted at 2024-05-01 08:10:59 on dist-test-slave-9cgk"
I20240501 08:10:59.635274 24667 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 08:10:59.638870 24681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:59.640244 24667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 08:10:59.640411 24667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "962160cac57040678e5cacd4b8b73c4b"
format_stamp: "Formatted at 2024-05-01 08:10:59 on dist-test-slave-9cgk"
I20240501 08:10:59.640611 24667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:59.663156 24667 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:59.664034 24667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:59.664312 24667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:59.665365 24667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:59.666911 24667 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:59.666999 24667 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:59.667081 24667 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:59.667133 24667 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:59.698179 24667 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.133:37341
I20240501 08:10:59.698220 24793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.133:37341 every 8 connection(s)
I20240501 08:10:59.699537 24667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 08:10:59.704821 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 24667
I20240501 08:10:59.705005 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 08:10:59.708403 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.134:0
--local_ip_for_outbound_sockets=127.21.84.134
--webserver_interface=127.21.84.134
--webserver_port=0
--tserver_master_addrs=127.21.84.190:43065
--builtin_ntp_servers=127.21.84.148:38087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:10:59.713052 24794 heartbeater.cc:347] Connected to a master server at 127.21.84.190:43065
I20240501 08:10:59.713377 24794 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:59.714262 24794 heartbeater.cc:511] Master 127.21.84.190:43065 requested a full tablet report, sending...
I20240501 08:10:59.715772 24087 ts_manager.cc:194] Registered new tserver with Master: 962160cac57040678e5cacd4b8b73c4b (127.21.84.133:37341)
I20240501 08:10:59.716948 24087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.133:59941
I20240501 08:10:59.777889 24534 heartbeater.cc:503] Master 127.21.84.190:43065 was elected leader, sending a full tablet report...
W20240501 08:11:00.047504 24797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:00.047839 24797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:00.047976 24797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:00.055431 24797 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:00.055600 24797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:00.055764 24797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.134
I20240501 08:11:00.064960 24797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:38087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.21.84.134
--webserver_port=0
--tserver_master_addrs=127.21.84.190:43065
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.134
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:00.066371 24797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:00.068183 24797 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:00.081177 24802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:00.081768 24803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:00.082724 24805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:00.083602 24797 server_base.cc:1034] running on GCE node
I20240501 08:11:00.084218 24797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:00.085204 24797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:00.086408 24797 hybrid_clock.cc:648] HybridClock initialized: now 1714551060086338 us; error 62 us; skew 500 ppm
I20240501 08:11:00.086748 24797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:00.089109 24797 webserver.cc:460] Webserver started at http://127.21.84.134:43761/ using document root <none> and password file <none>
I20240501 08:11:00.089820 24797 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:00.089917 24797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:00.090287 24797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:00.092808 24797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1"
format_stamp: "Formatted at 2024-05-01 08:11:00 on dist-test-slave-9cgk"
I20240501 08:11:00.093542 24797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1"
format_stamp: "Formatted at 2024-05-01 08:11:00 on dist-test-slave-9cgk"
I20240501 08:11:00.098829 24797 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20240501 08:11:00.102255 24811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:00.103655 24797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 08:11:00.103843 24797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1"
format_stamp: "Formatted at 2024-05-01 08:11:00 on dist-test-slave-9cgk"
I20240501 08:11:00.104046 24797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:00.118538 24797 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:00.119413 24797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:00.119699 24797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:00.120658 24797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:00.122160 24797 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:00.122246 24797 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:00.122370 24797 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:00.122439 24797 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:00.155066 24797 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.134:45565
I20240501 08:11:00.155164 24923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.134:45565 every 8 connection(s)
I20240501 08:11:00.156405 24797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 08:11:00.159078 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 24797
I20240501 08:11:00.159258 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 08:11:00.162875 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.135:0
--local_ip_for_outbound_sockets=127.21.84.135
--webserver_interface=127.21.84.135
--webserver_port=0
--tserver_master_addrs=127.21.84.190:43065
--builtin_ntp_servers=127.21.84.148:38087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:11:00.169803 24924 heartbeater.cc:347] Connected to a master server at 127.21.84.190:43065
I20240501 08:11:00.170181 24924 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:00.171044 24924 heartbeater.cc:511] Master 127.21.84.190:43065 requested a full tablet report, sending...
I20240501 08:11:00.172636 24087 ts_manager.cc:194] Registered new tserver with Master: 1ad8fc587a734cc48f7f49b00fc0c0a1 (127.21.84.134:45565)
I20240501 08:11:00.173383 24087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.134:58669
I20240501 08:11:00.258585 24664 heartbeater.cc:503] Master 127.21.84.190:43065 was elected leader, sending a full tablet report...
W20240501 08:11:00.499182 24927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:00.499526 24927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:00.499678 24927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:00.507102 24927 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:00.507272 24927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:00.507423 24927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.135
I20240501 08:11:00.516651 24927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:38087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.21.84.135
--webserver_port=0
--tserver_master_addrs=127.21.84.190:43065
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.135
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:00.518108 24927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:00.519814 24927 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:00.532155 24933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:00.532745 24932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:00.533779 24935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:00.534591 24927 server_base.cc:1034] running on GCE node
I20240501 08:11:00.535188 24927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:00.536187 24927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:00.537382 24927 hybrid_clock.cc:648] HybridClock initialized: now 1714551060537323 us; error 55 us; skew 500 ppm
I20240501 08:11:00.537711 24927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:00.540175 24927 webserver.cc:460] Webserver started at http://127.21.84.135:39105/ using document root <none> and password file <none>
I20240501 08:11:00.540968 24927 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:00.541083 24927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:00.541426 24927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:00.544277 24927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "8a102eff724b4f459bc91203802847fb"
format_stamp: "Formatted at 2024-05-01 08:11:00 on dist-test-slave-9cgk"
I20240501 08:11:00.545159 24927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "8a102eff724b4f459bc91203802847fb"
format_stamp: "Formatted at 2024-05-01 08:11:00 on dist-test-slave-9cgk"
I20240501 08:11:00.550702 24927 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240501 08:11:00.554405 24941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:00.555820 24927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 08:11:00.555969 24927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "8a102eff724b4f459bc91203802847fb"
format_stamp: "Formatted at 2024-05-01 08:11:00 on dist-test-slave-9cgk"
I20240501 08:11:00.556182 24927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:00.577098 24927 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:00.578029 24927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:00.578294 24927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:00.579216 24927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:00.580711 24927 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:00.580804 24927 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:00.580909 24927 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:00.580966 24927 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:00.612841 24927 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.135:35569
I20240501 08:11:00.612879 25053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.135:35569 every 8 connection(s)
I20240501 08:11:00.614326 24927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 08:11:00.622192 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 24927
I20240501 08:11:00.622396 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 08:11:00.627460 25054 heartbeater.cc:347] Connected to a master server at 127.21.84.190:43065
I20240501 08:11:00.627764 25054 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:00.628648 25054 heartbeater.cc:511] Master 127.21.84.190:43065 requested a full tablet report, sending...
I20240501 08:11:00.630137 24087 ts_manager.cc:194] Registered new tserver with Master: 8a102eff724b4f459bc91203802847fb (127.21.84.135:35569)
I20240501 08:11:00.631290 24087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.135:36339
I20240501 08:11:00.638684 21842 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 08:11:00.654930 24087 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43316:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 08:11:00.682701 24208 tablet_service.cc:1467] Processing CreateTablet for tablet b0c941c4d9ba4607b947765c218a1a60 (DEFAULT_TABLE table=TestTable [id=a447af67cd6347b1bcb9dbae4ec1aedf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:00.684038 24208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0c941c4d9ba4607b947765c218a1a60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:00.694753 25068 tablet_bootstrap.cc:492] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39: Bootstrap starting.
I20240501 08:11:00.697800 25068 tablet_bootstrap.cc:654] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:00.698863 25068 log.cc:824] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:00.701332 25068 tablet_bootstrap.cc:492] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39: No bootstrap required, opened a new log
I20240501 08:11:00.701640 25068 ts_tablet_manager.cc:1367] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 08:11:00.707870 25068 raft_consensus.cc:348] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } }
I20240501 08:11:00.708246 25068 raft_consensus.cc:374] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:00.708364 25068 raft_consensus.cc:729] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb59098726224e7bae383f5c11820b39, State: Initialized, Role: FOLLOWER
I20240501 08:11:00.709118 25068 consensus_queue.cc:260] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } }
I20240501 08:11:00.709326 25068 raft_consensus.cc:388] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:00.709450 25068 raft_consensus.cc:482] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:00.709600 25068 raft_consensus.cc:3037] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:00.711661 25068 raft_consensus.cc:504] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } }
I20240501 08:11:00.712261 25068 leader_election.cc:302] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb59098726224e7bae383f5c11820b39; no voters:
I20240501 08:11:00.712929 25068 leader_election.cc:288] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:00.713080 25070 raft_consensus.cc:2781] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:00.713740 25070 raft_consensus.cc:686] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Becoming Leader. State: Replica: fb59098726224e7bae383f5c11820b39, State: Running, Role: LEADER
I20240501 08:11:00.714339 25070 consensus_queue.cc:237] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } }
I20240501 08:11:00.714730 25068 ts_tablet_manager.cc:1398] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20240501 08:11:00.719552 24794 heartbeater.cc:503] Master 127.21.84.190:43065 was elected leader, sending a full tablet report...
I20240501 08:11:00.719491 24087 catalog_manager.cc:5484] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb59098726224e7bae383f5c11820b39 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "fb59098726224e7bae383f5c11820b39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:00.728742 21842 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 08:11:00.730787 21842 ts_itest-base.cc:233] Waiting for 1 tablets on tserver fb59098726224e7bae383f5c11820b39 to finish bootstrapping
I20240501 08:11:00.739157 21842 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 213ccbb05e6f44deb4a34d0a7041f7de to finish bootstrapping
I20240501 08:11:00.748278 21842 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f0c82b0e87d84b5eb3b2fd4f3e63ff1b to finish bootstrapping
I20240501 08:11:00.757592 21842 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 5ce52428c87b46369add4a8afaaef67c to finish bootstrapping
I20240501 08:11:00.766767 21842 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 962160cac57040678e5cacd4b8b73c4b to finish bootstrapping
I20240501 08:11:00.775400 21842 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 1ad8fc587a734cc48f7f49b00fc0c0a1 to finish bootstrapping
I20240501 08:11:00.784287 21842 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 8a102eff724b4f459bc91203802847fb to finish bootstrapping
I20240501 08:11:00.793154 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 24407
I20240501 08:11:00.794257 24529 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:00.874954 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 24407
I20240501 08:11:00.891120 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 24537
I20240501 08:11:00.892256 24659 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:00.974130 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 24537
I20240501 08:11:00.989763 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 24667
I20240501 08:11:00.990912 24789 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:01.072489 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 24667
I20240501 08:11:01.088078 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 24797
I20240501 08:11:01.089291 24919 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:01.172528 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 24797
I20240501 08:11:01.188347 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 24927
I20240501 08:11:01.189534 25049 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:01.271270 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 24927
I20240501 08:11:05.038879 24086 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43316:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:05.049309 24208 tablet_service.cc:1467] Processing CreateTablet for tablet 19eec94f19284c9e97efdbe449a5238d (DEFAULT_TABLE table=rebalance_test_table_0 [id=a0b88ef77ce744fd84b0987b68eda81a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:05.049782 24208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19eec94f19284c9e97efdbe449a5238d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:05.055418 25093 tablet_bootstrap.cc:492] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39: Bootstrap starting.
I20240501 08:11:05.056780 25093 tablet_bootstrap.cc:654] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:05.057899 24338 tablet_service.cc:1467] Processing CreateTablet for tablet c4bf2462d8b949d6a2f734780fd0d746 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a0b88ef77ce744fd84b0987b68eda81a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:05.059171 25093 tablet_bootstrap.cc:492] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39: No bootstrap required, opened a new log
I20240501 08:11:05.059314 25093 ts_tablet_manager.cc:1367] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:11:05.059497 24338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4bf2462d8b949d6a2f734780fd0d746. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:05.060279 25093 raft_consensus.cc:348] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } }
I20240501 08:11:05.060460 25093 raft_consensus.cc:374] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:05.060554 25093 raft_consensus.cc:729] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb59098726224e7bae383f5c11820b39, State: Initialized, Role: FOLLOWER
I20240501 08:11:05.060792 25093 consensus_queue.cc:260] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } }
I20240501 08:11:05.060981 25093 raft_consensus.cc:388] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:05.061059 25093 raft_consensus.cc:482] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:05.061167 25093 raft_consensus.cc:3037] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:05.057850 24339 tablet_service.cc:1467] Processing CreateTablet for tablet 28f02b777fc8455893ccb4bd4d585952 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a0b88ef77ce744fd84b0987b68eda81a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:05.062502 24339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28f02b777fc8455893ccb4bd4d585952. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:05.063699 25093 raft_consensus.cc:504] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } }
I20240501 08:11:05.063935 25093 leader_election.cc:302] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb59098726224e7bae383f5c11820b39; no voters:
I20240501 08:11:05.064388 25093 leader_election.cc:288] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:05.064554 25095 raft_consensus.cc:2781] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:05.068449 25093 ts_tablet_manager.cc:1398] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39: Time spent starting tablet: real 0.009s user 0.005s sys 0.004s
I20240501 08:11:05.073052 25098 tablet_bootstrap.cc:492] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de: Bootstrap starting.
I20240501 08:11:05.073702 25095 raft_consensus.cc:686] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Becoming Leader. State: Replica: fb59098726224e7bae383f5c11820b39, State: Running, Role: LEADER
I20240501 08:11:05.074015 25095 consensus_queue.cc:237] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } }
I20240501 08:11:05.076447 24086 catalog_manager.cc:5484] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb59098726224e7bae383f5c11820b39 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "fb59098726224e7bae383f5c11820b39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:05.076751 25098 tablet_bootstrap.cc:654] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:05.077965 25098 log.cc:824] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:05.080807 25098 tablet_bootstrap.cc:492] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de: No bootstrap required, opened a new log
I20240501 08:11:05.081099 25098 ts_tablet_manager.cc:1367] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240501 08:11:05.087481 25098 raft_consensus.cc:348] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } }
I20240501 08:11:05.087814 25098 raft_consensus.cc:374] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:05.087952 25098 raft_consensus.cc:729] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213ccbb05e6f44deb4a34d0a7041f7de, State: Initialized, Role: FOLLOWER
I20240501 08:11:05.088594 25098 consensus_queue.cc:260] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } }
I20240501 08:11:05.088778 25098 raft_consensus.cc:388] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:05.088920 25098 raft_consensus.cc:482] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:05.089085 25098 raft_consensus.cc:3037] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:05.091221 25098 raft_consensus.cc:504] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } }
I20240501 08:11:05.091682 25098 leader_election.cc:302] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 213ccbb05e6f44deb4a34d0a7041f7de; no voters:
I20240501 08:11:05.092234 25098 leader_election.cc:288] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:05.092375 25102 raft_consensus.cc:2781] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:05.093025 25102 raft_consensus.cc:686] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Becoming Leader. State: Replica: 213ccbb05e6f44deb4a34d0a7041f7de, State: Running, Role: LEADER
I20240501 08:11:05.093994 25098 ts_tablet_manager.cc:1398] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20240501 08:11:05.093961 25102 consensus_queue.cc:237] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } }
I20240501 08:11:05.094360 25098 tablet_bootstrap.cc:492] T 28f02b777fc8455893ccb4bd4d585952 P 213ccbb05e6f44deb4a34d0a7041f7de: Bootstrap starting.
I20240501 08:11:05.095809 25098 tablet_bootstrap.cc:654] T 28f02b777fc8455893ccb4bd4d585952 P 213ccbb05e6f44deb4a34d0a7041f7de: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:05.098238 25098 tablet_bootstrap.cc:492] T 28f02b777fc8455893ccb4bd4d585952 P 213ccbb05e6f44deb4a34d0a7041f7de: No bootstrap required, opened a new log
I20240501 08:11:05.098382 25098 ts_tablet_manager.cc:1367] T 28f02b777fc8455893ccb4bd4d585952 P 213ccbb05e6f44deb4a34d0a7041f7de: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:11:05.098409 24086 catalog_manager.cc:5484] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de reported cstate change: term changed from 0 to 1, leader changed from <none> to 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130). New cstate: current_term: 1 leader_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:05.099282 25098 raft_consensus.cc:348] T 28f02b777fc8455893ccb4bd4d585952 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } }
I20240501 08:11:05.099406 25098 raft_consensus.cc:374] T 28f02b777fc8455893ccb4bd4d585952 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:05.099495 25098 raft_consensus.cc:729] T 28f02b777fc8455893ccb4bd4d585952 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213ccbb05e6f44deb4a34d0a7041f7de, State: Initialized, Role: FOLLOWER
I20240501 08:11:05.099710 25098 consensus_queue.cc:260] T 28f02b777fc8455893ccb4bd4d585952 P 213ccbb05e6f44deb4a34d0a7041f7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } }
I20240501 08:11:05.099884 25098 raft_consensus.cc:388] T 28f02b777fc8455893ccb4bd4d585952 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:05.099956 25098 raft_consensus.cc:482] T 28f02b777fc8455893ccb4bd4d585952 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:05.100057 25098 raft_consensus.cc:3037] T 28f02b777fc8455893ccb4bd4d585952 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:05.102349 25098 raft_consensus.cc:504] T 28f02b777fc8455893ccb4bd4d585952 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } }
I20240501 08:11:05.102609 25098 leader_election.cc:302] T 28f02b777fc8455893ccb4bd4d585952 P 213ccbb05e6f44deb4a34d0a7041f7de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 213ccbb05e6f44deb4a34d0a7041f7de; no voters:
I20240501 08:11:05.102792 25098 leader_election.cc:288] T 28f02b777fc8455893ccb4bd4d585952 P 213ccbb05e6f44deb4a34d0a7041f7de [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:05.102847 25102 raft_consensus.cc:2781] T 28f02b777fc8455893ccb4bd4d585952 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:05.103008 25102 raft_consensus.cc:686] T 28f02b777fc8455893ccb4bd4d585952 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Becoming Leader. State: Replica: 213ccbb05e6f44deb4a34d0a7041f7de, State: Running, Role: LEADER
I20240501 08:11:05.103315 25098 ts_tablet_manager.cc:1398] T 28f02b777fc8455893ccb4bd4d585952 P 213ccbb05e6f44deb4a34d0a7041f7de: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 08:11:05.103235 25102 consensus_queue.cc:237] T 28f02b777fc8455893ccb4bd4d585952 P 213ccbb05e6f44deb4a34d0a7041f7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } }
I20240501 08:11:05.105065 24086 catalog_manager.cc:5484] T 28f02b777fc8455893ccb4bd4d585952 P 213ccbb05e6f44deb4a34d0a7041f7de reported cstate change: term changed from 0 to 1, leader changed from <none> to 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130). New cstate: current_term: 1 leader_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } health_report { overall_health: HEALTHY } } }
W20240501 08:11:05.450095 25109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:05.450423 25109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:05.458097 25109 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 147.38 ms
time per row: 0.0736901 ms
I20240501 08:11:05.751776 24079 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43316:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:05.760270 24208 tablet_service.cc:1467] Processing CreateTablet for tablet 476e8260712e414ab298ea94d413044e (DEFAULT_TABLE table=rebalance_test_table_1 [id=60c95048bf784e6ab27a6c2db06a3f90]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:05.760454 24339 tablet_service.cc:1467] Processing CreateTablet for tablet a331a1e85ff94e5fa93b42583f1668d5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=60c95048bf784e6ab27a6c2db06a3f90]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:05.760848 24339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a331a1e85ff94e5fa93b42583f1668d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:05.760885 24208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 476e8260712e414ab298ea94d413044e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:05.761001 24207 tablet_service.cc:1467] Processing CreateTablet for tablet abc7faa250d14cecb417da206b1927a3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=60c95048bf784e6ab27a6c2db06a3f90]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:05.761436 24207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abc7faa250d14cecb417da206b1927a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:05.766630 25156 tablet_bootstrap.cc:492] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39: Bootstrap starting.
I20240501 08:11:05.767580 25157 tablet_bootstrap.cc:492] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de: Bootstrap starting.
I20240501 08:11:05.768311 25156 tablet_bootstrap.cc:654] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:05.768958 25157 tablet_bootstrap.cc:654] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:05.770390 25156 tablet_bootstrap.cc:492] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39: No bootstrap required, opened a new log
I20240501 08:11:05.770547 25156 ts_tablet_manager.cc:1367] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:11:05.771029 25157 tablet_bootstrap.cc:492] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de: No bootstrap required, opened a new log
I20240501 08:11:05.771144 25157 ts_tablet_manager.cc:1367] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 08:11:05.771167 25156 raft_consensus.cc:348] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } }
I20240501 08:11:05.771328 25156 raft_consensus.cc:374] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:05.771414 25156 raft_consensus.cc:729] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb59098726224e7bae383f5c11820b39, State: Initialized, Role: FOLLOWER
I20240501 08:11:05.771656 25156 consensus_queue.cc:260] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } }
I20240501 08:11:05.771811 25156 raft_consensus.cc:388] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:05.771775 25157 raft_consensus.cc:348] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } }
I20240501 08:11:05.771885 25156 raft_consensus.cc:482] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:05.771948 25157 raft_consensus.cc:374] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:05.771981 25156 raft_consensus.cc:3037] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:05.772032 25157 raft_consensus.cc:729] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213ccbb05e6f44deb4a34d0a7041f7de, State: Initialized, Role: FOLLOWER
I20240501 08:11:05.772255 25157 consensus_queue.cc:260] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } }
I20240501 08:11:05.772399 25157 raft_consensus.cc:388] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:05.772478 25157 raft_consensus.cc:482] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:05.772586 25157 raft_consensus.cc:3037] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:05.774140 25156 raft_consensus.cc:504] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } }
I20240501 08:11:05.774354 25156 leader_election.cc:302] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb59098726224e7bae383f5c11820b39; no voters:
I20240501 08:11:05.774519 25156 leader_election.cc:288] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:05.774619 25145 raft_consensus.cc:2781] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:05.774775 25157 raft_consensus.cc:504] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } }
I20240501 08:11:05.775106 25157 leader_election.cc:302] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 213ccbb05e6f44deb4a34d0a7041f7de; no voters:
I20240501 08:11:05.775113 25145 raft_consensus.cc:686] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Becoming Leader. State: Replica: fb59098726224e7bae383f5c11820b39, State: Running, Role: LEADER
I20240501 08:11:05.775331 25157 leader_election.cc:288] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:05.775475 25145 consensus_queue.cc:237] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } }
I20240501 08:11:05.775779 25157 ts_tablet_manager.cc:1398] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20240501 08:11:05.776175 25143 raft_consensus.cc:2781] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:05.781227 25156 ts_tablet_manager.cc:1398] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39: Time spent starting tablet: real 0.011s user 0.004s sys 0.007s
I20240501 08:11:05.781049 24083 catalog_manager.cc:5484] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb59098726224e7bae383f5c11820b39 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "fb59098726224e7bae383f5c11820b39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:05.781497 25156 tablet_bootstrap.cc:492] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39: Bootstrap starting.
I20240501 08:11:05.782208 25143 raft_consensus.cc:686] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Becoming Leader. State: Replica: 213ccbb05e6f44deb4a34d0a7041f7de, State: Running, Role: LEADER
I20240501 08:11:05.782509 25143 consensus_queue.cc:237] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } }
I20240501 08:11:05.783896 25156 tablet_bootstrap.cc:654] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:05.784871 24079 catalog_manager.cc:5484] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de reported cstate change: term changed from 0 to 1, leader changed from <none> to 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130). New cstate: current_term: 1 leader_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:05.788224 25156 tablet_bootstrap.cc:492] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39: No bootstrap required, opened a new log
I20240501 08:11:05.788396 25156 ts_tablet_manager.cc:1367] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20240501 08:11:05.788967 25156 raft_consensus.cc:348] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } }
I20240501 08:11:05.789227 25156 raft_consensus.cc:374] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:05.789333 25156 raft_consensus.cc:729] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb59098726224e7bae383f5c11820b39, State: Initialized, Role: FOLLOWER
I20240501 08:11:05.789572 25156 consensus_queue.cc:260] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } }
I20240501 08:11:05.789736 25156 raft_consensus.cc:388] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:05.789834 25156 raft_consensus.cc:482] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:05.789914 25156 raft_consensus.cc:3037] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:05.791724 25156 raft_consensus.cc:504] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } }
I20240501 08:11:05.791913 25156 leader_election.cc:302] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb59098726224e7bae383f5c11820b39; no voters:
I20240501 08:11:05.792061 25156 leader_election.cc:288] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:05.792176 25160 raft_consensus.cc:2781] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:05.792353 25160 raft_consensus.cc:686] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Becoming Leader. State: Replica: fb59098726224e7bae383f5c11820b39, State: Running, Role: LEADER
I20240501 08:11:05.792495 25156 ts_tablet_manager.cc:1398] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:11:05.792661 25160 consensus_queue.cc:237] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } }
I20240501 08:11:05.794907 24083 catalog_manager.cc:5484] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb59098726224e7bae383f5c11820b39 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "fb59098726224e7bae383f5c11820b39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } health_report { overall_health: HEALTHY } } }
W20240501 08:11:06.148332 25167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:06.148691 25167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:06.156769 25167 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 173.63 ms
time per row: 0.0868151 ms
I20240501 08:11:06.472743 24083 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43316:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:06.481612 24338 tablet_service.cc:1467] Processing CreateTablet for tablet 0705e5b48e84423fa252da5d62b3008b (DEFAULT_TABLE table=rebalance_test_table_2 [id=f425dec73cd9491b943f3f9897a84256]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:06.481612 24208 tablet_service.cc:1467] Processing CreateTablet for tablet c3651f31063f43dd90576200ec3acbe5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f425dec73cd9491b943f3f9897a84256]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:06.481634 24339 tablet_service.cc:1467] Processing CreateTablet for tablet 6305a4cfdfc3420594ff185a15723ad1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f425dec73cd9491b943f3f9897a84256]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:06.482105 24208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3651f31063f43dd90576200ec3acbe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:06.482134 24338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0705e5b48e84423fa252da5d62b3008b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:06.482334 24339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6305a4cfdfc3420594ff185a15723ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:06.487831 25214 tablet_bootstrap.cc:492] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de: Bootstrap starting.
I20240501 08:11:06.489050 25215 tablet_bootstrap.cc:492] T c3651f31063f43dd90576200ec3acbe5 P fb59098726224e7bae383f5c11820b39: Bootstrap starting.
I20240501 08:11:06.489683 25214 tablet_bootstrap.cc:654] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:06.490921 25215 tablet_bootstrap.cc:654] T c3651f31063f43dd90576200ec3acbe5 P fb59098726224e7bae383f5c11820b39: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:06.491638 25214 tablet_bootstrap.cc:492] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de: No bootstrap required, opened a new log
I20240501 08:11:06.491788 25214 ts_tablet_manager.cc:1367] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 08:11:06.492373 25214 raft_consensus.cc:348] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } }
I20240501 08:11:06.492524 25214 raft_consensus.cc:374] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:06.492620 25214 raft_consensus.cc:729] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213ccbb05e6f44deb4a34d0a7041f7de, State: Initialized, Role: FOLLOWER
I20240501 08:11:06.492689 25215 tablet_bootstrap.cc:492] T c3651f31063f43dd90576200ec3acbe5 P fb59098726224e7bae383f5c11820b39: No bootstrap required, opened a new log
I20240501 08:11:06.492842 25215 ts_tablet_manager.cc:1367] T c3651f31063f43dd90576200ec3acbe5 P fb59098726224e7bae383f5c11820b39: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:11:06.492980 25214 consensus_queue.cc:260] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } }
I20240501 08:11:06.493162 25214 raft_consensus.cc:388] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:06.493254 25214 raft_consensus.cc:482] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:06.493371 25214 raft_consensus.cc:3037] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:06.493520 25215 raft_consensus.cc:348] T c3651f31063f43dd90576200ec3acbe5 P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } }
I20240501 08:11:06.493723 25215 raft_consensus.cc:374] T c3651f31063f43dd90576200ec3acbe5 P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:06.493840 25215 raft_consensus.cc:729] T c3651f31063f43dd90576200ec3acbe5 P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb59098726224e7bae383f5c11820b39, State: Initialized, Role: FOLLOWER
I20240501 08:11:06.494091 25215 consensus_queue.cc:260] T c3651f31063f43dd90576200ec3acbe5 P fb59098726224e7bae383f5c11820b39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } }
I20240501 08:11:06.494316 25215 raft_consensus.cc:388] T c3651f31063f43dd90576200ec3acbe5 P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:06.494406 25215 raft_consensus.cc:482] T c3651f31063f43dd90576200ec3acbe5 P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:06.494532 25215 raft_consensus.cc:3037] T c3651f31063f43dd90576200ec3acbe5 P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:06.495584 25214 raft_consensus.cc:504] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } }
I20240501 08:11:06.495819 25214 leader_election.cc:302] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 213ccbb05e6f44deb4a34d0a7041f7de; no voters:
I20240501 08:11:06.495998 25214 leader_election.cc:288] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:06.496098 25205 raft_consensus.cc:2781] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:06.496549 25215 raft_consensus.cc:504] T c3651f31063f43dd90576200ec3acbe5 P fb59098726224e7bae383f5c11820b39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } }
I20240501 08:11:06.496798 25214 ts_tablet_manager.cc:1398] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20240501 08:11:06.496992 25215 leader_election.cc:302] T c3651f31063f43dd90576200ec3acbe5 P fb59098726224e7bae383f5c11820b39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb59098726224e7bae383f5c11820b39; no voters:
I20240501 08:11:06.497100 25214 tablet_bootstrap.cc:492] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de: Bootstrap starting.
I20240501 08:11:06.497253 25215 leader_election.cc:288] T c3651f31063f43dd90576200ec3acbe5 P fb59098726224e7bae383f5c11820b39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:06.497336 25201 raft_consensus.cc:2781] T c3651f31063f43dd90576200ec3acbe5 P fb59098726224e7bae383f5c11820b39 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:06.497505 25201 raft_consensus.cc:686] T c3651f31063f43dd90576200ec3acbe5 P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Becoming Leader. State: Replica: fb59098726224e7bae383f5c11820b39, State: Running, Role: LEADER
I20240501 08:11:06.497849 25201 consensus_queue.cc:237] T c3651f31063f43dd90576200ec3acbe5 P fb59098726224e7bae383f5c11820b39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } }
I20240501 08:11:06.498853 25214 tablet_bootstrap.cc:654] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:06.504807 25205 raft_consensus.cc:686] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Becoming Leader. State: Replica: 213ccbb05e6f44deb4a34d0a7041f7de, State: Running, Role: LEADER
I20240501 08:11:06.505446 25205 consensus_queue.cc:237] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } }
I20240501 08:11:06.507733 25214 tablet_bootstrap.cc:492] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de: No bootstrap required, opened a new log
I20240501 08:11:06.508052 25214 ts_tablet_manager.cc:1367] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.010s
I20240501 08:11:06.509081 25214 raft_consensus.cc:348] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } }
I20240501 08:11:06.509361 25214 raft_consensus.cc:374] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:06.509505 25214 raft_consensus.cc:729] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213ccbb05e6f44deb4a34d0a7041f7de, State: Initialized, Role: FOLLOWER
I20240501 08:11:06.509827 25214 consensus_queue.cc:260] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } }
I20240501 08:11:06.510025 25214 raft_consensus.cc:388] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:06.510133 25214 raft_consensus.cc:482] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:06.510236 25214 raft_consensus.cc:3037] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:06.511482 25215 ts_tablet_manager.cc:1398] T c3651f31063f43dd90576200ec3acbe5 P fb59098726224e7bae383f5c11820b39: Time spent starting tablet: real 0.019s user 0.013s sys 0.004s
I20240501 08:11:06.511562 24081 catalog_manager.cc:5484] T c3651f31063f43dd90576200ec3acbe5 P fb59098726224e7bae383f5c11820b39 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb59098726224e7bae383f5c11820b39 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "fb59098726224e7bae383f5c11820b39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:06.513651 25214 raft_consensus.cc:504] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } }
I20240501 08:11:06.514137 25214 leader_election.cc:302] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 213ccbb05e6f44deb4a34d0a7041f7de; no voters:
I20240501 08:11:06.514122 24083 catalog_manager.cc:5484] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de reported cstate change: term changed from 0 to 1, leader changed from <none> to 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130). New cstate: current_term: 1 leader_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:06.514382 25214 leader_election.cc:288] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:06.514674 25219 raft_consensus.cc:2781] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:06.514832 25219 raft_consensus.cc:686] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Becoming Leader. State: Replica: 213ccbb05e6f44deb4a34d0a7041f7de, State: Running, Role: LEADER
I20240501 08:11:06.514840 25214 ts_tablet_manager.cc:1398] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20240501 08:11:06.515156 25219 consensus_queue.cc:237] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } }
I20240501 08:11:06.518976 24081 catalog_manager.cc:5484] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de reported cstate change: term changed from 0 to 1, leader changed from <none> to 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130). New cstate: current_term: 1 leader_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } health_report { overall_health: HEALTHY } } }
W20240501 08:11:06.862646 25225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:06.862973 25225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:06.870277 25225 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 131.39 ms
time per row: 0.0656951 ms
I20240501 08:11:07.143043 24087 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43316:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:07.151316 24208 tablet_service.cc:1467] Processing CreateTablet for tablet c5b68189903e46e48fde14764c024bff (DEFAULT_TABLE table=rebalance_test_table_3 [id=c0139c2c078545309a3a5eef71a69567]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:07.151441 24338 tablet_service.cc:1467] Processing CreateTablet for tablet dc01c06c62234c8e96be1da7795149c4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c0139c2c078545309a3a5eef71a69567]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:07.151814 24338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc01c06c62234c8e96be1da7795149c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:07.151813 24208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5b68189903e46e48fde14764c024bff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:07.151942 24207 tablet_service.cc:1467] Processing CreateTablet for tablet 9b065f9074a74fd9945f22596e7594db (DEFAULT_TABLE table=rebalance_test_table_3 [id=c0139c2c078545309a3a5eef71a69567]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:07.152266 24207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b065f9074a74fd9945f22596e7594db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:07.158411 25267 tablet_bootstrap.cc:492] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39: Bootstrap starting.
I20240501 08:11:07.158670 25268 tablet_bootstrap.cc:492] T dc01c06c62234c8e96be1da7795149c4 P 213ccbb05e6f44deb4a34d0a7041f7de: Bootstrap starting.
I20240501 08:11:07.160367 25268 tablet_bootstrap.cc:654] T dc01c06c62234c8e96be1da7795149c4 P 213ccbb05e6f44deb4a34d0a7041f7de: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:07.160367 25267 tablet_bootstrap.cc:654] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:07.162390 25268 tablet_bootstrap.cc:492] T dc01c06c62234c8e96be1da7795149c4 P 213ccbb05e6f44deb4a34d0a7041f7de: No bootstrap required, opened a new log
I20240501 08:11:07.162390 25267 tablet_bootstrap.cc:492] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39: No bootstrap required, opened a new log
I20240501 08:11:07.162542 25268 ts_tablet_manager.cc:1367] T dc01c06c62234c8e96be1da7795149c4 P 213ccbb05e6f44deb4a34d0a7041f7de: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:11:07.162541 25267 ts_tablet_manager.cc:1367] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:11:07.163314 25267 raft_consensus.cc:348] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } }
I20240501 08:11:07.163314 25268 raft_consensus.cc:348] T dc01c06c62234c8e96be1da7795149c4 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } }
I20240501 08:11:07.163558 25267 raft_consensus.cc:374] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:07.163556 25268 raft_consensus.cc:374] T dc01c06c62234c8e96be1da7795149c4 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:07.163652 25268 raft_consensus.cc:729] T dc01c06c62234c8e96be1da7795149c4 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213ccbb05e6f44deb4a34d0a7041f7de, State: Initialized, Role: FOLLOWER
I20240501 08:11:07.163699 25267 raft_consensus.cc:729] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb59098726224e7bae383f5c11820b39, State: Initialized, Role: FOLLOWER
I20240501 08:11:07.163957 25267 consensus_queue.cc:260] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } }
I20240501 08:11:07.163975 25268 consensus_queue.cc:260] T dc01c06c62234c8e96be1da7795149c4 P 213ccbb05e6f44deb4a34d0a7041f7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } }
I20240501 08:11:07.164180 25267 raft_consensus.cc:388] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:07.164182 25268 raft_consensus.cc:388] T dc01c06c62234c8e96be1da7795149c4 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:07.164275 25268 raft_consensus.cc:482] T dc01c06c62234c8e96be1da7795149c4 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:07.164276 25267 raft_consensus.cc:482] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:07.164387 25268 raft_consensus.cc:3037] T dc01c06c62234c8e96be1da7795149c4 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:07.164387 25267 raft_consensus.cc:3037] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:07.167152 25267 raft_consensus.cc:504] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } }
I20240501 08:11:07.167361 25268 raft_consensus.cc:504] T dc01c06c62234c8e96be1da7795149c4 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } }
I20240501 08:11:07.167491 25267 leader_election.cc:302] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb59098726224e7bae383f5c11820b39; no voters:
I20240501 08:11:07.167614 25268 leader_election.cc:302] T dc01c06c62234c8e96be1da7795149c4 P 213ccbb05e6f44deb4a34d0a7041f7de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 213ccbb05e6f44deb4a34d0a7041f7de; no voters:
I20240501 08:11:07.167711 25267 leader_election.cc:288] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:07.167814 25268 leader_election.cc:288] T dc01c06c62234c8e96be1da7795149c4 P 213ccbb05e6f44deb4a34d0a7041f7de [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:07.167922 25219 raft_consensus.cc:2781] T dc01c06c62234c8e96be1da7795149c4 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:07.168313 25268 ts_tablet_manager.cc:1398] T dc01c06c62234c8e96be1da7795149c4 P 213ccbb05e6f44deb4a34d0a7041f7de: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240501 08:11:07.167850 25220 raft_consensus.cc:2781] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:07.168512 25219 raft_consensus.cc:686] T dc01c06c62234c8e96be1da7795149c4 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Becoming Leader. State: Replica: 213ccbb05e6f44deb4a34d0a7041f7de, State: Running, Role: LEADER
I20240501 08:11:07.168780 25219 consensus_queue.cc:237] T dc01c06c62234c8e96be1da7795149c4 P 213ccbb05e6f44deb4a34d0a7041f7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } }
I20240501 08:11:07.169152 25267 ts_tablet_manager.cc:1398] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240501 08:11:07.169425 25267 tablet_bootstrap.cc:492] T c5b68189903e46e48fde14764c024bff P fb59098726224e7bae383f5c11820b39: Bootstrap starting.
I20240501 08:11:07.170864 25220 raft_consensus.cc:686] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Becoming Leader. State: Replica: fb59098726224e7bae383f5c11820b39, State: Running, Role: LEADER
I20240501 08:11:07.171217 25220 consensus_queue.cc:237] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } }
I20240501 08:11:07.171528 25267 tablet_bootstrap.cc:654] T c5b68189903e46e48fde14764c024bff P fb59098726224e7bae383f5c11820b39: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:07.172551 24087 catalog_manager.cc:5484] T dc01c06c62234c8e96be1da7795149c4 P 213ccbb05e6f44deb4a34d0a7041f7de reported cstate change: term changed from 0 to 1, leader changed from <none> to 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130). New cstate: current_term: 1 leader_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:07.173751 25267 tablet_bootstrap.cc:492] T c5b68189903e46e48fde14764c024bff P fb59098726224e7bae383f5c11820b39: No bootstrap required, opened a new log
I20240501 08:11:07.173909 25267 ts_tablet_manager.cc:1367] T c5b68189903e46e48fde14764c024bff P fb59098726224e7bae383f5c11820b39: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 08:11:07.174647 25267 raft_consensus.cc:348] T c5b68189903e46e48fde14764c024bff P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } }
I20240501 08:11:07.174865 25267 raft_consensus.cc:374] T c5b68189903e46e48fde14764c024bff P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:07.174959 25267 raft_consensus.cc:729] T c5b68189903e46e48fde14764c024bff P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb59098726224e7bae383f5c11820b39, State: Initialized, Role: FOLLOWER
I20240501 08:11:07.175092 24081 catalog_manager.cc:5484] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb59098726224e7bae383f5c11820b39 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "fb59098726224e7bae383f5c11820b39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:07.175184 25267 consensus_queue.cc:260] T c5b68189903e46e48fde14764c024bff P fb59098726224e7bae383f5c11820b39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } }
I20240501 08:11:07.175354 25267 raft_consensus.cc:388] T c5b68189903e46e48fde14764c024bff P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:07.175449 25267 raft_consensus.cc:482] T c5b68189903e46e48fde14764c024bff P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:07.175551 25267 raft_consensus.cc:3037] T c5b68189903e46e48fde14764c024bff P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:07.178767 25267 raft_consensus.cc:504] T c5b68189903e46e48fde14764c024bff P fb59098726224e7bae383f5c11820b39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } }
I20240501 08:11:07.179018 25267 leader_election.cc:302] T c5b68189903e46e48fde14764c024bff P fb59098726224e7bae383f5c11820b39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb59098726224e7bae383f5c11820b39; no voters:
I20240501 08:11:07.179210 25267 leader_election.cc:288] T c5b68189903e46e48fde14764c024bff P fb59098726224e7bae383f5c11820b39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:07.179294 25273 raft_consensus.cc:2781] T c5b68189903e46e48fde14764c024bff P fb59098726224e7bae383f5c11820b39 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:07.179474 25273 raft_consensus.cc:686] T c5b68189903e46e48fde14764c024bff P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Becoming Leader. State: Replica: fb59098726224e7bae383f5c11820b39, State: Running, Role: LEADER
I20240501 08:11:07.179617 25267 ts_tablet_manager.cc:1398] T c5b68189903e46e48fde14764c024bff P fb59098726224e7bae383f5c11820b39: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240501 08:11:07.179788 25273 consensus_queue.cc:237] T c5b68189903e46e48fde14764c024bff P fb59098726224e7bae383f5c11820b39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } }
I20240501 08:11:07.182157 24087 catalog_manager.cc:5484] T c5b68189903e46e48fde14764c024bff P fb59098726224e7bae383f5c11820b39 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb59098726224e7bae383f5c11820b39 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "fb59098726224e7bae383f5c11820b39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } health_report { overall_health: HEALTHY } } }
W20240501 08:11:07.521412 25278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:07.521796 25278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:07.529497 25278 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 139.212 ms
time per row: 0.0696062 ms
I20240501 08:11:07.809757 24081 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43316:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:07.817816 24338 tablet_service.cc:1467] Processing CreateTablet for tablet 158f407899b6440cb33d57025ab56873 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d5585afc570243fbbbb154585b168608]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:07.817967 24339 tablet_service.cc:1467] Processing CreateTablet for tablet 89a8cf2415454cb7b9157703f72913f4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d5585afc570243fbbbb154585b168608]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:07.818184 24208 tablet_service.cc:1467] Processing CreateTablet for tablet 49cdeb66bf204db3ac2e6c32f0705382 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d5585afc570243fbbbb154585b168608]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:07.818341 24338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 158f407899b6440cb33d57025ab56873. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:07.818560 24339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89a8cf2415454cb7b9157703f72913f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:07.818579 24208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49cdeb66bf204db3ac2e6c32f0705382. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:07.823763 25322 tablet_bootstrap.cc:492] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de: Bootstrap starting.
I20240501 08:11:07.825181 25323 tablet_bootstrap.cc:492] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39: Bootstrap starting.
I20240501 08:11:07.825683 25322 tablet_bootstrap.cc:654] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:07.826697 25323 tablet_bootstrap.cc:654] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:07.827610 25322 tablet_bootstrap.cc:492] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de: No bootstrap required, opened a new log
I20240501 08:11:07.827855 25322 ts_tablet_manager.cc:1367] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 08:11:07.828500 25322 raft_consensus.cc:348] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } }
I20240501 08:11:07.828653 25322 raft_consensus.cc:374] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:07.828725 25323 tablet_bootstrap.cc:492] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39: No bootstrap required, opened a new log
I20240501 08:11:07.828845 25322 raft_consensus.cc:729] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213ccbb05e6f44deb4a34d0a7041f7de, State: Initialized, Role: FOLLOWER
I20240501 08:11:07.828872 25323 ts_tablet_manager.cc:1367] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 08:11:07.829077 25322 consensus_queue.cc:260] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } }
I20240501 08:11:07.829228 25322 raft_consensus.cc:388] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:07.829304 25322 raft_consensus.cc:482] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:07.829406 25322 raft_consensus.cc:3037] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:07.829490 25323 raft_consensus.cc:348] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } }
I20240501 08:11:07.829640 25323 raft_consensus.cc:374] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:07.829726 25323 raft_consensus.cc:729] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb59098726224e7bae383f5c11820b39, State: Initialized, Role: FOLLOWER
I20240501 08:11:07.829962 25323 consensus_queue.cc:260] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } }
I20240501 08:11:07.830149 25323 raft_consensus.cc:388] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:07.830224 25323 raft_consensus.cc:482] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:07.830319 25323 raft_consensus.cc:3037] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:07.831449 25322 raft_consensus.cc:504] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } }
I20240501 08:11:07.831677 25322 leader_election.cc:302] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 213ccbb05e6f44deb4a34d0a7041f7de; no voters:
I20240501 08:11:07.831872 25322 leader_election.cc:288] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:07.832155 25315 raft_consensus.cc:2781] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:07.832204 25323 raft_consensus.cc:504] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } }
I20240501 08:11:07.832423 25323 leader_election.cc:302] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb59098726224e7bae383f5c11820b39; no voters:
I20240501 08:11:07.832597 25322 ts_tablet_manager.cc:1398] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
I20240501 08:11:07.832837 25322 tablet_bootstrap.cc:492] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de: Bootstrap starting.
I20240501 08:11:07.834205 25315 raft_consensus.cc:686] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Becoming Leader. State: Replica: 213ccbb05e6f44deb4a34d0a7041f7de, State: Running, Role: LEADER
I20240501 08:11:07.834352 25322 tablet_bootstrap.cc:654] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:07.834499 25315 consensus_queue.cc:237] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } }
I20240501 08:11:07.837148 25323 leader_election.cc:288] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:07.837219 25273 raft_consensus.cc:2781] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:07.837226 24084 catalog_manager.cc:5484] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de reported cstate change: term changed from 0 to 1, leader changed from <none> to 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130). New cstate: current_term: 1 leader_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:07.837724 25323 ts_tablet_manager.cc:1398] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39: Time spent starting tablet: real 0.009s user 0.000s sys 0.005s
I20240501 08:11:07.837862 25322 tablet_bootstrap.cc:492] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de: No bootstrap required, opened a new log
I20240501 08:11:07.837957 25273 raft_consensus.cc:686] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Becoming Leader. State: Replica: fb59098726224e7bae383f5c11820b39, State: Running, Role: LEADER
I20240501 08:11:07.838074 25322 ts_tablet_manager.cc:1367] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 08:11:07.838215 25273 consensus_queue.cc:237] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } }
I20240501 08:11:07.838850 25322 raft_consensus.cc:348] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } }
I20240501 08:11:07.839048 25322 raft_consensus.cc:374] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:07.839140 25322 raft_consensus.cc:729] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213ccbb05e6f44deb4a34d0a7041f7de, State: Initialized, Role: FOLLOWER
I20240501 08:11:07.839442 25322 consensus_queue.cc:260] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } }
I20240501 08:11:07.839653 25322 raft_consensus.cc:388] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:07.839764 25322 raft_consensus.cc:482] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:07.839888 25322 raft_consensus.cc:3037] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:07.842191 24084 catalog_manager.cc:5484] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb59098726224e7bae383f5c11820b39 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "fb59098726224e7bae383f5c11820b39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:07.844259 25322 raft_consensus.cc:504] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } }
I20240501 08:11:07.844558 25322 leader_election.cc:302] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 213ccbb05e6f44deb4a34d0a7041f7de; no voters:
I20240501 08:11:07.844837 25315 raft_consensus.cc:2781] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:07.845018 25315 raft_consensus.cc:686] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Becoming Leader. State: Replica: 213ccbb05e6f44deb4a34d0a7041f7de, State: Running, Role: LEADER
I20240501 08:11:07.845268 25315 consensus_queue.cc:237] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } }
I20240501 08:11:07.846045 25322 leader_election.cc:288] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:07.846621 25322 ts_tablet_manager.cc:1398] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240501 08:11:07.847767 24081 catalog_manager.cc:5484] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de reported cstate change: term changed from 0 to 1, leader changed from <none> to 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130). New cstate: current_term: 1 leader_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } health_report { overall_health: HEALTHY } } }
W20240501 08:11:08.189921 25332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:08.190376 25332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:08.197924 25332 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 145.057 ms
time per row: 0.0725287 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 08:11:09.404299 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.131:35031
--local_ip_for_outbound_sockets=127.21.84.131
--tserver_master_addrs=127.21.84.190:43065
--webserver_port=38437
--webserver_interface=127.21.84.131
--builtin_ntp_servers=127.21.84.148:38087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 08:11:09.738574 25369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:09.738945 25369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:09.739095 25369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:09.746919 25369 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:09.747085 25369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:09.747251 25369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.131
I20240501 08:11:09.756675 25369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:38087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.131:35031
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.21.84.131
--webserver_port=38437
--tserver_master_addrs=127.21.84.190:43065
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.131
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:09.758150 25369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:09.759881 25369 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:09.772401 25374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:09.773145 25375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:09.773873 25377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:09.774737 25369 server_base.cc:1034] running on GCE node
I20240501 08:11:09.775413 25369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:09.776422 25369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:09.777633 25369 hybrid_clock.cc:648] HybridClock initialized: now 1714551069777585 us; error 49 us; skew 500 ppm
I20240501 08:11:09.777976 25369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:09.780462 25369 webserver.cc:460] Webserver started at http://127.21.84.131:38437/ using document root <none> and password file <none>
I20240501 08:11:09.781212 25369 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:09.781342 25369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:09.786973 25369 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20240501 08:11:09.790073 25383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:09.791409 25369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 08:11:09.791590 25369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b"
format_stamp: "Formatted at 2024-05-01 08:10:58 on dist-test-slave-9cgk"
I20240501 08:11:09.792433 25369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:09.820376 25369 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:09.821224 25369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:09.821516 25369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:09.822494 25369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:09.824090 25369 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:09.824163 25369 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:09.824285 25369 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:09.824342 25369 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:09.857496 25369 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.131:35031
I20240501 08:11:09.857553 25495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.131:35031 every 8 connection(s)
I20240501 08:11:09.859079 25369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 08:11:09.864365 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 25369
I20240501 08:11:09.865207 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.132:43389
--local_ip_for_outbound_sockets=127.21.84.132
--tserver_master_addrs=127.21.84.190:43065
--webserver_port=40905
--webserver_interface=127.21.84.132
--builtin_ntp_servers=127.21.84.148:38087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:11:09.873903 25496 heartbeater.cc:347] Connected to a master server at 127.21.84.190:43065
I20240501 08:11:09.874313 25496 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:09.875478 25496 heartbeater.cc:511] Master 127.21.84.190:43065 requested a full tablet report, sending...
I20240501 08:11:09.877403 24083 ts_manager.cc:194] Re-registered known tserver with Master: f0c82b0e87d84b5eb3b2fd4f3e63ff1b (127.21.84.131:35031)
I20240501 08:11:09.878302 24083 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.131:52651
W20240501 08:11:10.210891 25500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:10.211205 25500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:10.211297 25500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:10.219036 25500 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:10.219175 25500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:10.219339 25500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.132
I20240501 08:11:10.228866 25500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:38087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.132:43389
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.21.84.132
--webserver_port=40905
--tserver_master_addrs=127.21.84.190:43065
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.132
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:10.230299 25500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:10.232158 25500 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:10.245431 25505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:10.245925 25506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:10.246690 25508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:10.251518 25500 server_base.cc:1034] running on GCE node
I20240501 08:11:10.252228 25500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:10.253247 25500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:10.254446 25500 hybrid_clock.cc:648] HybridClock initialized: now 1714551070254385 us; error 52 us; skew 500 ppm
I20240501 08:11:10.254839 25500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:10.257117 25500 webserver.cc:460] Webserver started at http://127.21.84.132:40905/ using document root <none> and password file <none>
I20240501 08:11:10.257858 25500 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:10.258009 25500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:10.263460 25500 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20240501 08:11:10.266499 25514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:10.267921 25500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240501 08:11:10.268097 25500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "5ce52428c87b46369add4a8afaaef67c"
format_stamp: "Formatted at 2024-05-01 08:10:59 on dist-test-slave-9cgk"
I20240501 08:11:10.268972 25500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:10.315348 25500 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:10.316249 25500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:10.316609 25500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:10.317600 25500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:10.319216 25500 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:10.319307 25500 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:10.319442 25500 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:10.319500 25500 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:10.352067 25500 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.132:43389
I20240501 08:11:10.352114 25626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.132:43389 every 8 connection(s)
I20240501 08:11:10.353523 25500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 08:11:10.357605 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 25500
I20240501 08:11:10.358348 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.133:37341
--local_ip_for_outbound_sockets=127.21.84.133
--tserver_master_addrs=127.21.84.190:43065
--webserver_port=32863
--webserver_interface=127.21.84.133
--builtin_ntp_servers=127.21.84.148:38087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:11:10.366950 25627 heartbeater.cc:347] Connected to a master server at 127.21.84.190:43065
I20240501 08:11:10.367269 25627 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:10.368252 25627 heartbeater.cc:511] Master 127.21.84.190:43065 requested a full tablet report, sending...
I20240501 08:11:10.369863 24083 ts_manager.cc:194] Re-registered known tserver with Master: 5ce52428c87b46369add4a8afaaef67c (127.21.84.132:43389)
I20240501 08:11:10.370704 24083 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.132:51143
W20240501 08:11:10.688560 25630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:10.688887 25630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:10.689056 25630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:10.696411 25630 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:10.696578 25630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:10.696702 25630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.133
I20240501 08:11:10.705466 25630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:38087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.133:37341
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.21.84.133
--webserver_port=32863
--tserver_master_addrs=127.21.84.190:43065
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.133
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:10.706880 25630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:10.708564 25630 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:10.721297 25635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:10.721890 25636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:10.722821 25638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:10.723754 25630 server_base.cc:1034] running on GCE node
I20240501 08:11:10.724357 25630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:10.725322 25630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:10.726536 25630 hybrid_clock.cc:648] HybridClock initialized: now 1714551070726478 us; error 45 us; skew 500 ppm
I20240501 08:11:10.726908 25630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:10.733034 25630 webserver.cc:460] Webserver started at http://127.21.84.133:32863/ using document root <none> and password file <none>
I20240501 08:11:10.733861 25630 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:10.734027 25630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:10.739526 25630 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20240501 08:11:10.742503 25644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:10.744012 25630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240501 08:11:10.744191 25630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "962160cac57040678e5cacd4b8b73c4b"
format_stamp: "Formatted at 2024-05-01 08:10:59 on dist-test-slave-9cgk"
I20240501 08:11:10.745074 25630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:10.775627 25630 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:10.776479 25630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:10.776762 25630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:10.777740 25630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:10.779256 25630 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:10.779335 25630 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:10.779459 25630 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:10.779512 25630 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:10.812871 25630 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.133:37341
I20240501 08:11:10.812916 25756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.133:37341 every 8 connection(s)
I20240501 08:11:10.814416 25630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 08:11:10.818599 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 25630
I20240501 08:11:10.819312 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.134:45565
--local_ip_for_outbound_sockets=127.21.84.134
--tserver_master_addrs=127.21.84.190:43065
--webserver_port=43761
--webserver_interface=127.21.84.134
--builtin_ntp_servers=127.21.84.148:38087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:11:10.828683 25757 heartbeater.cc:347] Connected to a master server at 127.21.84.190:43065
I20240501 08:11:10.829038 25757 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:10.829911 25757 heartbeater.cc:511] Master 127.21.84.190:43065 requested a full tablet report, sending...
I20240501 08:11:10.831517 24083 ts_manager.cc:194] Re-registered known tserver with Master: 962160cac57040678e5cacd4b8b73c4b (127.21.84.133:37341)
I20240501 08:11:10.832878 24083 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.133:54857
I20240501 08:11:10.881834 25496 heartbeater.cc:503] Master 127.21.84.190:43065 was elected leader, sending a full tablet report...
W20240501 08:11:11.157146 25760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:11.157459 25760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:11.157547 25760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:11.165100 25760 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:11.165243 25760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:11.165351 25760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.134
I20240501 08:11:11.174804 25760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:38087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.134:45565
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.21.84.134
--webserver_port=43761
--tserver_master_addrs=127.21.84.190:43065
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.134
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:11.176129 25760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:11.177841 25760 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:11.190443 25765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:11.191063 25768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:11.192101 25766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:11.192983 25760 server_base.cc:1034] running on GCE node
I20240501 08:11:11.193652 25760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:11.194671 25760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:11.195894 25760 hybrid_clock.cc:648] HybridClock initialized: now 1714551071195846 us; error 46 us; skew 500 ppm
I20240501 08:11:11.196226 25760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:11.198696 25760 webserver.cc:460] Webserver started at http://127.21.84.134:43761/ using document root <none> and password file <none>
I20240501 08:11:11.199438 25760 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:11.199575 25760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:11.205324 25760 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 08:11:11.208384 25774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:11.209863 25760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:11.210079 25760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1"
format_stamp: "Formatted at 2024-05-01 08:11:00 on dist-test-slave-9cgk"
I20240501 08:11:11.210968 25760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:11.235893 25760 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:11.236795 25760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:11.237090 25760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:11.238123 25760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:11.240115 25760 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:11.240211 25760 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:11.240350 25760 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:11.240406 25760 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:11.272709 25760 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.134:45565
I20240501 08:11:11.272743 25886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.134:45565 every 8 connection(s)
I20240501 08:11:11.274406 25760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 08:11:11.280342 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 25760
I20240501 08:11:11.281131 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.135:35569
--local_ip_for_outbound_sockets=127.21.84.135
--tserver_master_addrs=127.21.84.190:43065
--webserver_port=39105
--webserver_interface=127.21.84.135
--builtin_ntp_servers=127.21.84.148:38087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:11:11.288925 25887 heartbeater.cc:347] Connected to a master server at 127.21.84.190:43065
I20240501 08:11:11.289350 25887 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:11.290468 25887 heartbeater.cc:511] Master 127.21.84.190:43065 requested a full tablet report, sending...
I20240501 08:11:11.292320 24083 ts_manager.cc:194] Re-registered known tserver with Master: 1ad8fc587a734cc48f7f49b00fc0c0a1 (127.21.84.134:45565)
I20240501 08:11:11.293521 24083 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.134:44057
I20240501 08:11:11.373561 25627 heartbeater.cc:503] Master 127.21.84.190:43065 was elected leader, sending a full tablet report...
W20240501 08:11:11.622227 25890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:11.622543 25890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:11.622671 25890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:11.630571 25890 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:11.630717 25890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:11.630893 25890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.135
I20240501 08:11:11.640337 25890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:38087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.135:35569
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.21.84.135
--webserver_port=39105
--tserver_master_addrs=127.21.84.190:43065
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.135
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:11.641741 25890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:11.643718 25890 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:11.657321 25896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:11.658213 25895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:11.658921 25898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:11.659709 25890 server_base.cc:1034] running on GCE node
I20240501 08:11:11.660344 25890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:11.661401 25890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:11.662603 25890 hybrid_clock.cc:648] HybridClock initialized: now 1714551071662548 us; error 42 us; skew 500 ppm
I20240501 08:11:11.662940 25890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:11.665189 25890 webserver.cc:460] Webserver started at http://127.21.84.135:39105/ using document root <none> and password file <none>
I20240501 08:11:11.666002 25890 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:11.666134 25890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:11.671624 25890 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20240501 08:11:11.674638 25904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:11.676038 25890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:11:11.676213 25890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "8a102eff724b4f459bc91203802847fb"
format_stamp: "Formatted at 2024-05-01 08:11:00 on dist-test-slave-9cgk"
I20240501 08:11:11.677073 25890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:11.699791 25890 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:11.700630 25890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:11.700903 25890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:11.701823 25890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:11.703351 25890 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:11.703420 25890 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:11.703545 25890 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:11.703593 25890 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:11.736994 26016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.135:35569 every 8 connection(s)
I20240501 08:11:11.736941 25890 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.135:35569
I20240501 08:11:11.738665 25890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 08:11:11.742517 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 25890
I20240501 08:11:11.754501 26017 heartbeater.cc:347] Connected to a master server at 127.21.84.190:43065
I20240501 08:11:11.754825 26017 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:11.755714 26017 heartbeater.cc:511] Master 127.21.84.190:43065 requested a full tablet report, sending...
I20240501 08:11:11.757200 24083 ts_manager.cc:194] Re-registered known tserver with Master: 8a102eff724b4f459bc91203802847fb (127.21.84.135:35569)
I20240501 08:11:11.758005 24083 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.135:48551
I20240501 08:11:11.836163 25757 heartbeater.cc:503] Master 127.21.84.190:43065 was elected leader, sending a full tablet report...
I20240501 08:11:12.259052 24339 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:11:12.277786 25822 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:12.280539 25952 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:12.288832 25562 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:12.298218 25887 heartbeater.cc:503] Master 127.21.84.190:43065 was elected leader, sending a full tablet report...
I20240501 08:11:12.317785 25692 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:12.319061 25431 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:12.320972 24208 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:11:12.548576 25822 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:12.549660 25952 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:12.552644 25562 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:12.556624 24339 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:11:12.560153 25692 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:12.577296 25431 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:12.591322 24208 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:11:12.670037 24359 consensus_queue.cc:237] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "5ce52428c87b46369add4a8afaaef67c" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 43389 } attrs { promote: true } }
I20240501 08:11:12.672629 26107 raft_consensus.cc:2932] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5ce52428c87b46369add4a8afaaef67c (127.21.84.132) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "5ce52428c87b46369add4a8afaaef67c" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 43389 } attrs { promote: true } } }
I20240501 08:11:12.679672 24087 catalog_manager.cc:5484] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de reported cstate change: config changed from index -1 to 4, NON_VOTER 5ce52428c87b46369add4a8afaaef67c (127.21.84.132) added. New cstate: current_term: 1 leader_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ce52428c87b46369add4a8afaaef67c" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 43389 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:11:12.683365 24293 consensus_peers.cc:485] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de -> Peer 5ce52428c87b46369add4a8afaaef67c (127.21.84.132:43389): Couldn't send request to peer 5ce52428c87b46369add4a8afaaef67c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0705e5b48e84423fa252da5d62b3008b. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:12.692056 24228 consensus_queue.cc:237] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 35031 } attrs { promote: true } }
I20240501 08:11:12.694636 26120 raft_consensus.cc:2932] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f0c82b0e87d84b5eb3b2fd4f3e63ff1b (127.21.84.131) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 35031 } attrs { promote: true } } }
I20240501 08:11:12.700960 24087 catalog_manager.cc:5484] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39 reported cstate change: config changed from index -1 to 4, NON_VOTER f0c82b0e87d84b5eb3b2fd4f3e63ff1b (127.21.84.131) added. New cstate: current_term: 1 leader_uuid: "fb59098726224e7bae383f5c11820b39" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 35031 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:11:12.704761 24161 consensus_peers.cc:485] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39 -> Peer f0c82b0e87d84b5eb3b2fd4f3e63ff1b (127.21.84.131:35031): Couldn't send request to peer f0c82b0e87d84b5eb3b2fd4f3e63ff1b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 476e8260712e414ab298ea94d413044e. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:12.712514 24359 consensus_queue.cc:237] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 45565 } attrs { promote: true } }
I20240501 08:11:12.714473 26106 raft_consensus.cc:2932] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1ad8fc587a734cc48f7f49b00fc0c0a1 (127.21.84.134) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 45565 } attrs { promote: true } } }
I20240501 08:11:12.720633 24087 catalog_manager.cc:5484] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de reported cstate change: config changed from index -1 to 4, NON_VOTER 1ad8fc587a734cc48f7f49b00fc0c0a1 (127.21.84.134) added. New cstate: current_term: 1 leader_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 45565 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:11:12.724164 24292 consensus_peers.cc:485] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de -> Peer 1ad8fc587a734cc48f7f49b00fc0c0a1 (127.21.84.134:45565): Couldn't send request to peer 1ad8fc587a734cc48f7f49b00fc0c0a1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 89a8cf2415454cb7b9157703f72913f4. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:12.731626 24228 consensus_queue.cc:237] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "8a102eff724b4f459bc91203802847fb" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 35569 } attrs { promote: true } }
I20240501 08:11:12.733446 26120 raft_consensus.cc:2932] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8a102eff724b4f459bc91203802847fb (127.21.84.135) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "8a102eff724b4f459bc91203802847fb" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 35569 } attrs { promote: true } } }
I20240501 08:11:12.738946 24087 catalog_manager.cc:5484] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39 reported cstate change: config changed from index -1 to 4, NON_VOTER 8a102eff724b4f459bc91203802847fb (127.21.84.135) added. New cstate: current_term: 1 leader_uuid: "fb59098726224e7bae383f5c11820b39" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a102eff724b4f459bc91203802847fb" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 35569 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:11:12.742489 24161 consensus_peers.cc:485] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39 -> Peer 8a102eff724b4f459bc91203802847fb (127.21.84.135:35569): Couldn't send request to peer 8a102eff724b4f459bc91203802847fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: abc7faa250d14cecb417da206b1927a3. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:12.759480 24228 consensus_queue.cc:237] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "962160cac57040678e5cacd4b8b73c4b" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 37341 } attrs { promote: true } }
I20240501 08:11:12.760461 26017 heartbeater.cc:503] Master 127.21.84.190:43065 was elected leader, sending a full tablet report...
I20240501 08:11:12.761927 26120 raft_consensus.cc:2932] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 962160cac57040678e5cacd4b8b73c4b (127.21.84.133) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "962160cac57040678e5cacd4b8b73c4b" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 37341 } attrs { promote: true } } }
I20240501 08:11:12.767692 24087 catalog_manager.cc:5484] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39 reported cstate change: config changed from index -1 to 4, NON_VOTER 962160cac57040678e5cacd4b8b73c4b (127.21.84.133) added. New cstate: current_term: 1 leader_uuid: "fb59098726224e7bae383f5c11820b39" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "962160cac57040678e5cacd4b8b73c4b" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 37341 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:11:12.769569 24164 consensus_peers.cc:485] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39 -> Peer 962160cac57040678e5cacd4b8b73c4b (127.21.84.133:37341): Couldn't send request to peer 962160cac57040678e5cacd4b8b73c4b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b065f9074a74fd9945f22596e7594db. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:12.779147 24228 consensus_queue.cc:237] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "5ce52428c87b46369add4a8afaaef67c" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 43389 } attrs { promote: true } }
I20240501 08:11:12.781417 26120 raft_consensus.cc:2932] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5ce52428c87b46369add4a8afaaef67c (127.21.84.132) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "5ce52428c87b46369add4a8afaaef67c" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 43389 } attrs { promote: true } } }
I20240501 08:11:12.787335 24081 catalog_manager.cc:5484] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39 reported cstate change: config changed from index -1 to 4, NON_VOTER 5ce52428c87b46369add4a8afaaef67c (127.21.84.132) added. New cstate: current_term: 1 leader_uuid: "fb59098726224e7bae383f5c11820b39" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ce52428c87b46369add4a8afaaef67c" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 43389 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:11:12.788172 24162 consensus_peers.cc:485] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39 -> Peer 5ce52428c87b46369add4a8afaaef67c (127.21.84.132:43389): Couldn't send request to peer 5ce52428c87b46369add4a8afaaef67c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19eec94f19284c9e97efdbe449a5238d. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:12.796881 24359 consensus_queue.cc:237] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 35031 } attrs { promote: true } }
I20240501 08:11:12.799165 26106 raft_consensus.cc:2932] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f0c82b0e87d84b5eb3b2fd4f3e63ff1b (127.21.84.131) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 35031 } attrs { promote: true } } }
I20240501 08:11:12.803831 24081 catalog_manager.cc:5484] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de reported cstate change: config changed from index -1 to 4, NON_VOTER f0c82b0e87d84b5eb3b2fd4f3e63ff1b (127.21.84.131) added. New cstate: current_term: 1 leader_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 35031 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:11:12.804527 24292 consensus_peers.cc:485] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de -> Peer f0c82b0e87d84b5eb3b2fd4f3e63ff1b (127.21.84.131:35031): Couldn't send request to peer f0c82b0e87d84b5eb3b2fd4f3e63ff1b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 158f407899b6440cb33d57025ab56873. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:12.814860 24359 consensus_queue.cc:237] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 45565 } attrs { promote: true } }
I20240501 08:11:12.816730 26106 raft_consensus.cc:2932] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1ad8fc587a734cc48f7f49b00fc0c0a1 (127.21.84.134) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 45565 } attrs { promote: true } } }
W20240501 08:11:12.817196 24292 consensus_peers.cc:485] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de -> Peer 1ad8fc587a734cc48f7f49b00fc0c0a1 (127.21.84.134:45565): Couldn't send request to peer 1ad8fc587a734cc48f7f49b00fc0c0a1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6305a4cfdfc3420594ff185a15723ad1. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:12.822312 24081 catalog_manager.cc:5484] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de reported cstate change: config changed from index -1 to 4, NON_VOTER 1ad8fc587a734cc48f7f49b00fc0c0a1 (127.21.84.134) added. New cstate: current_term: 1 leader_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 45565 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:12.833150 24359 consensus_queue.cc:237] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "8a102eff724b4f459bc91203802847fb" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 35569 } attrs { promote: true } }
I20240501 08:11:12.834997 26106 raft_consensus.cc:2932] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8a102eff724b4f459bc91203802847fb (127.21.84.135) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "8a102eff724b4f459bc91203802847fb" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 35569 } attrs { promote: true } } }
I20240501 08:11:12.842552 24081 catalog_manager.cc:5484] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de reported cstate change: config changed from index -1 to 4, NON_VOTER 8a102eff724b4f459bc91203802847fb (127.21.84.135) added. New cstate: current_term: 1 leader_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a102eff724b4f459bc91203802847fb" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 35569 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:11:12.843770 24292 consensus_peers.cc:485] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de -> Peer 8a102eff724b4f459bc91203802847fb (127.21.84.135:35569): Couldn't send request to peer 8a102eff724b4f459bc91203802847fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c4bf2462d8b949d6a2f734780fd0d746. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:12.861164 24228 consensus_queue.cc:237] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "962160cac57040678e5cacd4b8b73c4b" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 37341 } attrs { promote: true } }
I20240501 08:11:12.862888 26119 raft_consensus.cc:2932] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 962160cac57040678e5cacd4b8b73c4b (127.21.84.133) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "962160cac57040678e5cacd4b8b73c4b" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 37341 } attrs { promote: true } } }
W20240501 08:11:12.863324 24164 consensus_peers.cc:485] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39 -> Peer 962160cac57040678e5cacd4b8b73c4b (127.21.84.133:37341): Couldn't send request to peer 962160cac57040678e5cacd4b8b73c4b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b0c941c4d9ba4607b947765c218a1a60. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:12.868275 24087 catalog_manager.cc:5484] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39 reported cstate change: config changed from index -1 to 2, NON_VOTER 962160cac57040678e5cacd4b8b73c4b (127.21.84.133) added. New cstate: current_term: 1 leader_uuid: "fb59098726224e7bae383f5c11820b39" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "962160cac57040678e5cacd4b8b73c4b" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 37341 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:13.110493 26221 ts_tablet_manager.cc:902] T 476e8260712e414ab298ea94d413044e P f0c82b0e87d84b5eb3b2fd4f3e63ff1b: Initiating tablet copy from peer fb59098726224e7bae383f5c11820b39 (127.21.84.129:44759)
I20240501 08:11:13.113112 26221 tablet_copy_client.cc:308] T 476e8260712e414ab298ea94d413044e P f0c82b0e87d84b5eb3b2fd4f3e63ff1b: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.129:44759
I20240501 08:11:13.120705 24248 tablet_copy_service.cc:140] P fb59098726224e7bae383f5c11820b39: Received BeginTabletCopySession request for tablet 476e8260712e414ab298ea94d413044e from peer f0c82b0e87d84b5eb3b2fd4f3e63ff1b ({username='slave'} at 127.21.84.131:57477)
I20240501 08:11:13.121042 24248 tablet_copy_service.cc:161] P fb59098726224e7bae383f5c11820b39: Beginning new tablet copy session on tablet 476e8260712e414ab298ea94d413044e from peer f0c82b0e87d84b5eb3b2fd4f3e63ff1b at {username='slave'} at 127.21.84.131:57477: session id = f0c82b0e87d84b5eb3b2fd4f3e63ff1b-476e8260712e414ab298ea94d413044e
I20240501 08:11:13.122872 26227 ts_tablet_manager.cc:902] T abc7faa250d14cecb417da206b1927a3 P 8a102eff724b4f459bc91203802847fb: Initiating tablet copy from peer fb59098726224e7bae383f5c11820b39 (127.21.84.129:44759)
I20240501 08:11:13.124727 26227 tablet_copy_client.cc:308] T abc7faa250d14cecb417da206b1927a3 P 8a102eff724b4f459bc91203802847fb: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.129:44759
I20240501 08:11:13.125095 24248 tablet_copy_source_session.cc:205] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:13.130893 26221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 476e8260712e414ab298ea94d413044e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:13.132158 24248 tablet_copy_service.cc:140] P fb59098726224e7bae383f5c11820b39: Received BeginTabletCopySession request for tablet abc7faa250d14cecb417da206b1927a3 from peer 8a102eff724b4f459bc91203802847fb ({username='slave'} at 127.21.84.135:51749)
I20240501 08:11:13.132311 24248 tablet_copy_service.cc:161] P fb59098726224e7bae383f5c11820b39: Beginning new tablet copy session on tablet abc7faa250d14cecb417da206b1927a3 from peer 8a102eff724b4f459bc91203802847fb at {username='slave'} at 127.21.84.135:51749: session id = 8a102eff724b4f459bc91203802847fb-abc7faa250d14cecb417da206b1927a3
I20240501 08:11:13.134434 24248 tablet_copy_source_session.cc:205] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:13.138518 26227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abc7faa250d14cecb417da206b1927a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:13.141350 26221 tablet_copy_client.cc:783] T 476e8260712e414ab298ea94d413044e P f0c82b0e87d84b5eb3b2fd4f3e63ff1b: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:13.141681 26221 tablet_copy_client.cc:647] T 476e8260712e414ab298ea94d413044e P f0c82b0e87d84b5eb3b2fd4f3e63ff1b: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:13.144464 26221 tablet_copy_client.cc:523] T 476e8260712e414ab298ea94d413044e P f0c82b0e87d84b5eb3b2fd4f3e63ff1b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:13.147531 26221 tablet_bootstrap.cc:492] T 476e8260712e414ab298ea94d413044e P f0c82b0e87d84b5eb3b2fd4f3e63ff1b: Bootstrap starting.
I20240501 08:11:13.149780 26227 tablet_copy_client.cc:783] T abc7faa250d14cecb417da206b1927a3 P 8a102eff724b4f459bc91203802847fb: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:13.150143 26227 tablet_copy_client.cc:647] T abc7faa250d14cecb417da206b1927a3 P 8a102eff724b4f459bc91203802847fb: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:13.152406 26227 tablet_copy_client.cc:523] T abc7faa250d14cecb417da206b1927a3 P 8a102eff724b4f459bc91203802847fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:13.155984 26227 tablet_bootstrap.cc:492] T abc7faa250d14cecb417da206b1927a3 P 8a102eff724b4f459bc91203802847fb: Bootstrap starting.
I20240501 08:11:13.157259 26221 log.cc:824] T 476e8260712e414ab298ea94d413044e P f0c82b0e87d84b5eb3b2fd4f3e63ff1b: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:13.168529 26227 log.cc:824] T abc7faa250d14cecb417da206b1927a3 P 8a102eff724b4f459bc91203802847fb: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:13.200289 26221 tablet_bootstrap.cc:492] T 476e8260712e414ab298ea94d413044e P f0c82b0e87d84b5eb3b2fd4f3e63ff1b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:13.200325 26250 ts_tablet_manager.cc:902] T 89a8cf2415454cb7b9157703f72913f4 P 1ad8fc587a734cc48f7f49b00fc0c0a1: Initiating tablet copy from peer 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130:37857)
I20240501 08:11:13.200888 26221 tablet_bootstrap.cc:492] T 476e8260712e414ab298ea94d413044e P f0c82b0e87d84b5eb3b2fd4f3e63ff1b: Bootstrap complete.
I20240501 08:11:13.201422 26221 ts_tablet_manager.cc:1367] T 476e8260712e414ab298ea94d413044e P f0c82b0e87d84b5eb3b2fd4f3e63ff1b: Time spent bootstrapping tablet: real 0.054s user 0.042s sys 0.009s
I20240501 08:11:13.213809 26221 raft_consensus.cc:348] T 476e8260712e414ab298ea94d413044e P f0c82b0e87d84b5eb3b2fd4f3e63ff1b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 35031 } attrs { promote: true } }
I20240501 08:11:13.214514 26221 raft_consensus.cc:729] T 476e8260712e414ab298ea94d413044e P f0c82b0e87d84b5eb3b2fd4f3e63ff1b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0c82b0e87d84b5eb3b2fd4f3e63ff1b, State: Initialized, Role: LEARNER
I20240501 08:11:13.215863 26221 consensus_queue.cc:260] T 476e8260712e414ab298ea94d413044e P f0c82b0e87d84b5eb3b2fd4f3e63ff1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 35031 } attrs { promote: true } }
I20240501 08:11:13.224834 26221 ts_tablet_manager.cc:1398] T 476e8260712e414ab298ea94d413044e P f0c82b0e87d84b5eb3b2fd4f3e63ff1b: Time spent starting tablet: real 0.023s user 0.020s sys 0.004s
I20240501 08:11:13.229288 24248 tablet_copy_service.cc:342] P fb59098726224e7bae383f5c11820b39: Request end of tablet copy session f0c82b0e87d84b5eb3b2fd4f3e63ff1b-476e8260712e414ab298ea94d413044e received from {username='slave'} at 127.21.84.131:57477
I20240501 08:11:13.229481 24248 tablet_copy_service.cc:433] P fb59098726224e7bae383f5c11820b39: ending tablet copy session f0c82b0e87d84b5eb3b2fd4f3e63ff1b-476e8260712e414ab298ea94d413044e on tablet 476e8260712e414ab298ea94d413044e with peer f0c82b0e87d84b5eb3b2fd4f3e63ff1b
I20240501 08:11:13.234833 26227 tablet_bootstrap.cc:492] T abc7faa250d14cecb417da206b1927a3 P 8a102eff724b4f459bc91203802847fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:13.235433 26227 tablet_bootstrap.cc:492] T abc7faa250d14cecb417da206b1927a3 P 8a102eff724b4f459bc91203802847fb: Bootstrap complete.
I20240501 08:11:13.235881 26227 ts_tablet_manager.cc:1367] T abc7faa250d14cecb417da206b1927a3 P 8a102eff724b4f459bc91203802847fb: Time spent bootstrapping tablet: real 0.080s user 0.072s sys 0.005s
I20240501 08:11:13.236789 26250 tablet_copy_client.cc:308] T 89a8cf2415454cb7b9157703f72913f4 P 1ad8fc587a734cc48f7f49b00fc0c0a1: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.130:37857
I20240501 08:11:13.244570 26227 raft_consensus.cc:348] T abc7faa250d14cecb417da206b1927a3 P 8a102eff724b4f459bc91203802847fb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "8a102eff724b4f459bc91203802847fb" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 35569 } attrs { promote: true } }
I20240501 08:11:13.245095 26227 raft_consensus.cc:729] T abc7faa250d14cecb417da206b1927a3 P 8a102eff724b4f459bc91203802847fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8a102eff724b4f459bc91203802847fb, State: Initialized, Role: LEARNER
I20240501 08:11:13.245750 26227 consensus_queue.cc:260] T abc7faa250d14cecb417da206b1927a3 P 8a102eff724b4f459bc91203802847fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "8a102eff724b4f459bc91203802847fb" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 35569 } attrs { promote: true } }
I20240501 08:11:13.251884 24379 tablet_copy_service.cc:140] P 213ccbb05e6f44deb4a34d0a7041f7de: Received BeginTabletCopySession request for tablet 89a8cf2415454cb7b9157703f72913f4 from peer 1ad8fc587a734cc48f7f49b00fc0c0a1 ({username='slave'} at 127.21.84.134:44235)
I20240501 08:11:13.252149 24379 tablet_copy_service.cc:161] P 213ccbb05e6f44deb4a34d0a7041f7de: Beginning new tablet copy session on tablet 89a8cf2415454cb7b9157703f72913f4 from peer 1ad8fc587a734cc48f7f49b00fc0c0a1 at {username='slave'} at 127.21.84.134:44235: session id = 1ad8fc587a734cc48f7f49b00fc0c0a1-89a8cf2415454cb7b9157703f72913f4
I20240501 08:11:13.253545 26227 ts_tablet_manager.cc:1398] T abc7faa250d14cecb417da206b1927a3 P 8a102eff724b4f459bc91203802847fb: Time spent starting tablet: real 0.018s user 0.012s sys 0.004s
I20240501 08:11:13.255811 24379 tablet_copy_source_session.cc:205] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:13.257148 24248 tablet_copy_service.cc:342] P fb59098726224e7bae383f5c11820b39: Request end of tablet copy session 8a102eff724b4f459bc91203802847fb-abc7faa250d14cecb417da206b1927a3 received from {username='slave'} at 127.21.84.135:51749
I20240501 08:11:13.257290 24248 tablet_copy_service.cc:433] P fb59098726224e7bae383f5c11820b39: ending tablet copy session 8a102eff724b4f459bc91203802847fb-abc7faa250d14cecb417da206b1927a3 on tablet abc7faa250d14cecb417da206b1927a3 with peer 8a102eff724b4f459bc91203802847fb
I20240501 08:11:13.261698 26250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89a8cf2415454cb7b9157703f72913f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:13.274623 26269 ts_tablet_manager.cc:902] T 0705e5b48e84423fa252da5d62b3008b P 5ce52428c87b46369add4a8afaaef67c: Initiating tablet copy from peer 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130:37857)
I20240501 08:11:13.285004 26250 tablet_copy_client.cc:783] T 89a8cf2415454cb7b9157703f72913f4 P 1ad8fc587a734cc48f7f49b00fc0c0a1: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:13.285387 26250 tablet_copy_client.cc:647] T 89a8cf2415454cb7b9157703f72913f4 P 1ad8fc587a734cc48f7f49b00fc0c0a1: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:13.288653 26250 tablet_copy_client.cc:523] T 89a8cf2415454cb7b9157703f72913f4 P 1ad8fc587a734cc48f7f49b00fc0c0a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:13.292125 26250 tablet_bootstrap.cc:492] T 89a8cf2415454cb7b9157703f72913f4 P 1ad8fc587a734cc48f7f49b00fc0c0a1: Bootstrap starting.
I20240501 08:11:13.297814 26269 tablet_copy_client.cc:308] T 0705e5b48e84423fa252da5d62b3008b P 5ce52428c87b46369add4a8afaaef67c: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.130:37857
I20240501 08:11:13.313727 24379 tablet_copy_service.cc:140] P 213ccbb05e6f44deb4a34d0a7041f7de: Received BeginTabletCopySession request for tablet 0705e5b48e84423fa252da5d62b3008b from peer 5ce52428c87b46369add4a8afaaef67c ({username='slave'} at 127.21.84.132:37865)
I20240501 08:11:13.314035 24379 tablet_copy_service.cc:161] P 213ccbb05e6f44deb4a34d0a7041f7de: Beginning new tablet copy session on tablet 0705e5b48e84423fa252da5d62b3008b from peer 5ce52428c87b46369add4a8afaaef67c at {username='slave'} at 127.21.84.132:37865: session id = 5ce52428c87b46369add4a8afaaef67c-0705e5b48e84423fa252da5d62b3008b
I20240501 08:11:13.317190 26250 log.cc:824] T 89a8cf2415454cb7b9157703f72913f4 P 1ad8fc587a734cc48f7f49b00fc0c0a1: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:13.320032 24379 tablet_copy_source_session.cc:205] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:13.327311 26269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0705e5b48e84423fa252da5d62b3008b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:13.344173 26288 ts_tablet_manager.cc:902] T 9b065f9074a74fd9945f22596e7594db P 962160cac57040678e5cacd4b8b73c4b: Initiating tablet copy from peer fb59098726224e7bae383f5c11820b39 (127.21.84.129:44759)
I20240501 08:11:13.362329 26221 ts_tablet_manager.cc:902] T 158f407899b6440cb33d57025ab56873 P f0c82b0e87d84b5eb3b2fd4f3e63ff1b: Initiating tablet copy from peer 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130:37857)
I20240501 08:11:13.366596 26227 ts_tablet_manager.cc:902] T c4bf2462d8b949d6a2f734780fd0d746 P 8a102eff724b4f459bc91203802847fb: Initiating tablet copy from peer 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130:37857)
I20240501 08:11:13.374078 26269 tablet_copy_client.cc:783] T 0705e5b48e84423fa252da5d62b3008b P 5ce52428c87b46369add4a8afaaef67c: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:13.374555 26269 tablet_copy_client.cc:647] T 0705e5b48e84423fa252da5d62b3008b P 5ce52428c87b46369add4a8afaaef67c: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:13.388695 26269 tablet_copy_client.cc:523] T 0705e5b48e84423fa252da5d62b3008b P 5ce52428c87b46369add4a8afaaef67c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:13.390612 26292 ts_tablet_manager.cc:902] T b0c941c4d9ba4607b947765c218a1a60 P 962160cac57040678e5cacd4b8b73c4b: Initiating tablet copy from peer fb59098726224e7bae383f5c11820b39 (127.21.84.129:44759)
I20240501 08:11:13.396103 26269 tablet_bootstrap.cc:492] T 0705e5b48e84423fa252da5d62b3008b P 5ce52428c87b46369add4a8afaaef67c: Bootstrap starting.
I20240501 08:11:13.398965 26298 ts_tablet_manager.cc:902] T 19eec94f19284c9e97efdbe449a5238d P 5ce52428c87b46369add4a8afaaef67c: Initiating tablet copy from peer fb59098726224e7bae383f5c11820b39 (127.21.84.129:44759)
I20240501 08:11:13.399019 26297 ts_tablet_manager.cc:902] T 6305a4cfdfc3420594ff185a15723ad1 P 1ad8fc587a734cc48f7f49b00fc0c0a1: Initiating tablet copy from peer 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130:37857)
I20240501 08:11:13.414214 26292 tablet_copy_client.cc:308] T b0c941c4d9ba4607b947765c218a1a60 P 962160cac57040678e5cacd4b8b73c4b: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.129:44759
I20240501 08:11:13.418705 26227 tablet_copy_client.cc:308] T c4bf2462d8b949d6a2f734780fd0d746 P 8a102eff724b4f459bc91203802847fb: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.130:37857
I20240501 08:11:13.423638 26298 tablet_copy_client.cc:308] T 19eec94f19284c9e97efdbe449a5238d P 5ce52428c87b46369add4a8afaaef67c: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.129:44759
I20240501 08:11:13.429811 24379 tablet_copy_service.cc:140] P 213ccbb05e6f44deb4a34d0a7041f7de: Received BeginTabletCopySession request for tablet c4bf2462d8b949d6a2f734780fd0d746 from peer 8a102eff724b4f459bc91203802847fb ({username='slave'} at 127.21.84.135:56429)
I20240501 08:11:13.430042 24379 tablet_copy_service.cc:161] P 213ccbb05e6f44deb4a34d0a7041f7de: Beginning new tablet copy session on tablet c4bf2462d8b949d6a2f734780fd0d746 from peer 8a102eff724b4f459bc91203802847fb at {username='slave'} at 127.21.84.135:56429: session id = 8a102eff724b4f459bc91203802847fb-c4bf2462d8b949d6a2f734780fd0d746
I20240501 08:11:13.432957 24379 tablet_copy_source_session.cc:205] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:13.434722 26227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4bf2462d8b949d6a2f734780fd0d746. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:13.442625 26288 tablet_copy_client.cc:308] T 9b065f9074a74fd9945f22596e7594db P 962160cac57040678e5cacd4b8b73c4b: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.129:44759
I20240501 08:11:13.451637 26250 tablet_bootstrap.cc:492] T 89a8cf2415454cb7b9157703f72913f4 P 1ad8fc587a734cc48f7f49b00fc0c0a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:13.451751 26227 tablet_copy_client.cc:783] T c4bf2462d8b949d6a2f734780fd0d746 P 8a102eff724b4f459bc91203802847fb: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:13.452044 26227 tablet_copy_client.cc:647] T c4bf2462d8b949d6a2f734780fd0d746 P 8a102eff724b4f459bc91203802847fb: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:13.452591 26250 tablet_bootstrap.cc:492] T 89a8cf2415454cb7b9157703f72913f4 P 1ad8fc587a734cc48f7f49b00fc0c0a1: Bootstrap complete.
I20240501 08:11:13.453166 26250 ts_tablet_manager.cc:1367] T 89a8cf2415454cb7b9157703f72913f4 P 1ad8fc587a734cc48f7f49b00fc0c0a1: Time spent bootstrapping tablet: real 0.161s user 0.059s sys 0.025s
I20240501 08:11:13.454478 26227 tablet_copy_client.cc:523] T c4bf2462d8b949d6a2f734780fd0d746 P 8a102eff724b4f459bc91203802847fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:13.458216 26227 tablet_bootstrap.cc:492] T c4bf2462d8b949d6a2f734780fd0d746 P 8a102eff724b4f459bc91203802847fb: Bootstrap starting.
I20240501 08:11:13.461205 26250 raft_consensus.cc:348] T 89a8cf2415454cb7b9157703f72913f4 P 1ad8fc587a734cc48f7f49b00fc0c0a1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 45565 } attrs { promote: true } }
I20240501 08:11:13.462021 26250 raft_consensus.cc:729] T 89a8cf2415454cb7b9157703f72913f4 P 1ad8fc587a734cc48f7f49b00fc0c0a1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ad8fc587a734cc48f7f49b00fc0c0a1, State: Initialized, Role: LEARNER
I20240501 08:11:13.462925 26250 consensus_queue.cc:260] T 89a8cf2415454cb7b9157703f72913f4 P 1ad8fc587a734cc48f7f49b00fc0c0a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 45565 } attrs { promote: true } }
I20240501 08:11:13.464584 24248 tablet_copy_service.cc:140] P fb59098726224e7bae383f5c11820b39: Received BeginTabletCopySession request for tablet 19eec94f19284c9e97efdbe449a5238d from peer 5ce52428c87b46369add4a8afaaef67c ({username='slave'} at 127.21.84.132:43125)
I20240501 08:11:13.464819 24248 tablet_copy_service.cc:161] P fb59098726224e7bae383f5c11820b39: Beginning new tablet copy session on tablet 19eec94f19284c9e97efdbe449a5238d from peer 5ce52428c87b46369add4a8afaaef67c at {username='slave'} at 127.21.84.132:43125: session id = 5ce52428c87b46369add4a8afaaef67c-19eec94f19284c9e97efdbe449a5238d
I20240501 08:11:13.468767 24248 tablet_copy_source_session.cc:205] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:13.470539 24247 tablet_copy_service.cc:140] P fb59098726224e7bae383f5c11820b39: Received BeginTabletCopySession request for tablet 9b065f9074a74fd9945f22596e7594db from peer 962160cac57040678e5cacd4b8b73c4b ({username='slave'} at 127.21.84.133:59095)
I20240501 08:11:13.470777 24247 tablet_copy_service.cc:161] P fb59098726224e7bae383f5c11820b39: Beginning new tablet copy session on tablet 9b065f9074a74fd9945f22596e7594db from peer 962160cac57040678e5cacd4b8b73c4b at {username='slave'} at 127.21.84.133:59095: session id = 962160cac57040678e5cacd4b8b73c4b-9b065f9074a74fd9945f22596e7594db
I20240501 08:11:13.475286 24247 tablet_copy_source_session.cc:205] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:13.477669 24248 tablet_copy_service.cc:140] P fb59098726224e7bae383f5c11820b39: Received BeginTabletCopySession request for tablet b0c941c4d9ba4607b947765c218a1a60 from peer 962160cac57040678e5cacd4b8b73c4b ({username='slave'} at 127.21.84.133:59095)
I20240501 08:11:13.477916 24248 tablet_copy_service.cc:161] P fb59098726224e7bae383f5c11820b39: Beginning new tablet copy session on tablet b0c941c4d9ba4607b947765c218a1a60 from peer 962160cac57040678e5cacd4b8b73c4b at {username='slave'} at 127.21.84.133:59095: session id = 962160cac57040678e5cacd4b8b73c4b-b0c941c4d9ba4607b947765c218a1a60
I20240501 08:11:13.481920 24248 tablet_copy_source_session.cc:205] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:13.482352 26288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b065f9074a74fd9945f22596e7594db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:13.483546 26292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0c941c4d9ba4607b947765c218a1a60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:13.487980 26297 tablet_copy_client.cc:308] T 6305a4cfdfc3420594ff185a15723ad1 P 1ad8fc587a734cc48f7f49b00fc0c0a1: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.130:37857
I20240501 08:11:13.488855 26298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19eec94f19284c9e97efdbe449a5238d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:13.488934 24379 tablet_copy_service.cc:140] P 213ccbb05e6f44deb4a34d0a7041f7de: Received BeginTabletCopySession request for tablet 6305a4cfdfc3420594ff185a15723ad1 from peer 1ad8fc587a734cc48f7f49b00fc0c0a1 ({username='slave'} at 127.21.84.134:44235)
I20240501 08:11:13.489205 24379 tablet_copy_service.cc:161] P 213ccbb05e6f44deb4a34d0a7041f7de: Beginning new tablet copy session on tablet 6305a4cfdfc3420594ff185a15723ad1 from peer 1ad8fc587a734cc48f7f49b00fc0c0a1 at {username='slave'} at 127.21.84.134:44235: session id = 1ad8fc587a734cc48f7f49b00fc0c0a1-6305a4cfdfc3420594ff185a15723ad1
I20240501 08:11:13.494117 24379 tablet_copy_source_session.cc:205] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:13.494372 26221 tablet_copy_client.cc:308] T 158f407899b6440cb33d57025ab56873 P f0c82b0e87d84b5eb3b2fd4f3e63ff1b: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.130:37857
I20240501 08:11:13.502889 26297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6305a4cfdfc3420594ff185a15723ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:13.503474 26298 tablet_copy_client.cc:783] T 19eec94f19284c9e97efdbe449a5238d P 5ce52428c87b46369add4a8afaaef67c: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:13.503952 26298 tablet_copy_client.cc:647] T 19eec94f19284c9e97efdbe449a5238d P 5ce52428c87b46369add4a8afaaef67c: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:13.506978 24379 tablet_copy_service.cc:140] P 213ccbb05e6f44deb4a34d0a7041f7de: Received BeginTabletCopySession request for tablet 158f407899b6440cb33d57025ab56873 from peer f0c82b0e87d84b5eb3b2fd4f3e63ff1b ({username='slave'} at 127.21.84.131:46057)
I20240501 08:11:13.507216 24379 tablet_copy_service.cc:161] P 213ccbb05e6f44deb4a34d0a7041f7de: Beginning new tablet copy session on tablet 158f407899b6440cb33d57025ab56873 from peer f0c82b0e87d84b5eb3b2fd4f3e63ff1b at {username='slave'} at 127.21.84.131:46057: session id = f0c82b0e87d84b5eb3b2fd4f3e63ff1b-158f407899b6440cb33d57025ab56873
I20240501 08:11:13.509891 26298 tablet_copy_client.cc:523] T 19eec94f19284c9e97efdbe449a5238d P 5ce52428c87b46369add4a8afaaef67c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:13.511301 24379 tablet_copy_source_session.cc:205] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:13.512794 26297 tablet_copy_client.cc:783] T 6305a4cfdfc3420594ff185a15723ad1 P 1ad8fc587a734cc48f7f49b00fc0c0a1: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:13.513109 26297 tablet_copy_client.cc:647] T 6305a4cfdfc3420594ff185a15723ad1 P 1ad8fc587a734cc48f7f49b00fc0c0a1: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:13.514117 26221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 158f407899b6440cb33d57025ab56873. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:13.517760 26298 tablet_bootstrap.cc:492] T 19eec94f19284c9e97efdbe449a5238d P 5ce52428c87b46369add4a8afaaef67c: Bootstrap starting.
I20240501 08:11:13.517879 26250 ts_tablet_manager.cc:1398] T 89a8cf2415454cb7b9157703f72913f4 P 1ad8fc587a734cc48f7f49b00fc0c0a1: Time spent starting tablet: real 0.065s user 0.034s sys 0.016s
I20240501 08:11:13.519153 24378 tablet_copy_service.cc:342] P 213ccbb05e6f44deb4a34d0a7041f7de: Request end of tablet copy session 1ad8fc587a734cc48f7f49b00fc0c0a1-89a8cf2415454cb7b9157703f72913f4 received from {username='slave'} at 127.21.84.134:44235
I20240501 08:11:13.519308 24378 tablet_copy_service.cc:433] P 213ccbb05e6f44deb4a34d0a7041f7de: ending tablet copy session 1ad8fc587a734cc48f7f49b00fc0c0a1-89a8cf2415454cb7b9157703f72913f4 on tablet 89a8cf2415454cb7b9157703f72913f4 with peer 1ad8fc587a734cc48f7f49b00fc0c0a1
I20240501 08:11:13.524641 26221 tablet_copy_client.cc:783] T 158f407899b6440cb33d57025ab56873 P f0c82b0e87d84b5eb3b2fd4f3e63ff1b: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:13.524895 26221 tablet_copy_client.cc:647] T 158f407899b6440cb33d57025ab56873 P f0c82b0e87d84b5eb3b2fd4f3e63ff1b: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:13.524978 26297 tablet_copy_client.cc:523] T 6305a4cfdfc3420594ff185a15723ad1 P 1ad8fc587a734cc48f7f49b00fc0c0a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:13.527918 26288 tablet_copy_client.cc:783] T 9b065f9074a74fd9945f22596e7594db P 962160cac57040678e5cacd4b8b73c4b: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:13.528223 26288 tablet_copy_client.cc:647] T 9b065f9074a74fd9945f22596e7594db P 962160cac57040678e5cacd4b8b73c4b: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:13.528605 26297 tablet_bootstrap.cc:492] T 6305a4cfdfc3420594ff185a15723ad1 P 1ad8fc587a734cc48f7f49b00fc0c0a1: Bootstrap starting.
I20240501 08:11:13.528563 25451 raft_consensus.cc:1203] T 476e8260712e414ab298ea94d413044e P f0c82b0e87d84b5eb3b2fd4f3e63ff1b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:11:13.529587 26292 tablet_copy_client.cc:783] T b0c941c4d9ba4607b947765c218a1a60 P 962160cac57040678e5cacd4b8b73c4b: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:13.529832 26292 tablet_copy_client.cc:647] T b0c941c4d9ba4607b947765c218a1a60 P 962160cac57040678e5cacd4b8b73c4b: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:13.531831 26221 tablet_copy_client.cc:523] T 158f407899b6440cb33d57025ab56873 P f0c82b0e87d84b5eb3b2fd4f3e63ff1b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:13.534780 26288 tablet_copy_client.cc:523] T 9b065f9074a74fd9945f22596e7594db P 962160cac57040678e5cacd4b8b73c4b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:13.536175 26221 tablet_bootstrap.cc:492] T 158f407899b6440cb33d57025ab56873 P f0c82b0e87d84b5eb3b2fd4f3e63ff1b: Bootstrap starting.
I20240501 08:11:13.539417 26288 tablet_bootstrap.cc:492] T 9b065f9074a74fd9945f22596e7594db P 962160cac57040678e5cacd4b8b73c4b: Bootstrap starting.
I20240501 08:11:13.550762 26292 tablet_copy_client.cc:523] T b0c941c4d9ba4607b947765c218a1a60 P 962160cac57040678e5cacd4b8b73c4b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:13.558974 26292 tablet_bootstrap.cc:492] T b0c941c4d9ba4607b947765c218a1a60 P 962160cac57040678e5cacd4b8b73c4b: Bootstrap starting.
I20240501 08:11:13.527091 26269 log.cc:824] T 0705e5b48e84423fa252da5d62b3008b P 5ce52428c87b46369add4a8afaaef67c: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:13.563115 25972 raft_consensus.cc:1203] T abc7faa250d14cecb417da206b1927a3 P 8a102eff724b4f459bc91203802847fb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:11:13.572223 26292 log.cc:824] T b0c941c4d9ba4607b947765c218a1a60 P 962160cac57040678e5cacd4b8b73c4b: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:13.590926 26292 tablet_bootstrap.cc:492] T b0c941c4d9ba4607b947765c218a1a60 P 962160cac57040678e5cacd4b8b73c4b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:13.591621 26292 tablet_bootstrap.cc:492] T b0c941c4d9ba4607b947765c218a1a60 P 962160cac57040678e5cacd4b8b73c4b: Bootstrap complete.
I20240501 08:11:13.592200 26292 ts_tablet_manager.cc:1367] T b0c941c4d9ba4607b947765c218a1a60 P 962160cac57040678e5cacd4b8b73c4b: Time spent bootstrapping tablet: real 0.033s user 0.008s sys 0.009s
I20240501 08:11:13.612675 26297 tablet_bootstrap.cc:492] T 6305a4cfdfc3420594ff185a15723ad1 P 1ad8fc587a734cc48f7f49b00fc0c0a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:13.612435 26292 raft_consensus.cc:348] T b0c941c4d9ba4607b947765c218a1a60 P 962160cac57040678e5cacd4b8b73c4b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "962160cac57040678e5cacd4b8b73c4b" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 37341 } attrs { promote: true } }
I20240501 08:11:13.613121 26297 tablet_bootstrap.cc:492] T 6305a4cfdfc3420594ff185a15723ad1 P 1ad8fc587a734cc48f7f49b00fc0c0a1: Bootstrap complete.
I20240501 08:11:13.613227 26292 raft_consensus.cc:729] T b0c941c4d9ba4607b947765c218a1a60 P 962160cac57040678e5cacd4b8b73c4b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 962160cac57040678e5cacd4b8b73c4b, State: Initialized, Role: LEARNER
I20240501 08:11:13.613399 26297 ts_tablet_manager.cc:1367] T 6305a4cfdfc3420594ff185a15723ad1 P 1ad8fc587a734cc48f7f49b00fc0c0a1: Time spent bootstrapping tablet: real 0.085s user 0.050s sys 0.014s
I20240501 08:11:13.614001 26292 consensus_queue.cc:260] T b0c941c4d9ba4607b947765c218a1a60 P 962160cac57040678e5cacd4b8b73c4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "962160cac57040678e5cacd4b8b73c4b" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 37341 } attrs { promote: true } }
I20240501 08:11:13.614391 26297 raft_consensus.cc:348] T 6305a4cfdfc3420594ff185a15723ad1 P 1ad8fc587a734cc48f7f49b00fc0c0a1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 45565 } attrs { promote: true } }
I20240501 08:11:13.614625 26297 raft_consensus.cc:729] T 6305a4cfdfc3420594ff185a15723ad1 P 1ad8fc587a734cc48f7f49b00fc0c0a1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ad8fc587a734cc48f7f49b00fc0c0a1, State: Initialized, Role: LEARNER
I20240501 08:11:13.614820 26297 consensus_queue.cc:260] T 6305a4cfdfc3420594ff185a15723ad1 P 1ad8fc587a734cc48f7f49b00fc0c0a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 45565 } attrs { promote: true } }
I20240501 08:11:13.618158 26297 ts_tablet_manager.cc:1398] T 6305a4cfdfc3420594ff185a15723ad1 P 1ad8fc587a734cc48f7f49b00fc0c0a1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 08:11:13.621071 24379 tablet_copy_service.cc:342] P 213ccbb05e6f44deb4a34d0a7041f7de: Request end of tablet copy session 1ad8fc587a734cc48f7f49b00fc0c0a1-6305a4cfdfc3420594ff185a15723ad1 received from {username='slave'} at 127.21.84.134:44235
I20240501 08:11:13.621243 24379 tablet_copy_service.cc:433] P 213ccbb05e6f44deb4a34d0a7041f7de: ending tablet copy session 1ad8fc587a734cc48f7f49b00fc0c0a1-6305a4cfdfc3420594ff185a15723ad1 on tablet 6305a4cfdfc3420594ff185a15723ad1 with peer 1ad8fc587a734cc48f7f49b00fc0c0a1
I20240501 08:11:13.623410 26227 tablet_bootstrap.cc:492] T c4bf2462d8b949d6a2f734780fd0d746 P 8a102eff724b4f459bc91203802847fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:13.623847 26227 tablet_bootstrap.cc:492] T c4bf2462d8b949d6a2f734780fd0d746 P 8a102eff724b4f459bc91203802847fb: Bootstrap complete.
I20240501 08:11:13.624099 26227 ts_tablet_manager.cc:1367] T c4bf2462d8b949d6a2f734780fd0d746 P 8a102eff724b4f459bc91203802847fb: Time spent bootstrapping tablet: real 0.166s user 0.049s sys 0.004s
I20240501 08:11:13.624840 26227 raft_consensus.cc:348] T c4bf2462d8b949d6a2f734780fd0d746 P 8a102eff724b4f459bc91203802847fb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "8a102eff724b4f459bc91203802847fb" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 35569 } attrs { promote: true } }
I20240501 08:11:13.625026 26227 raft_consensus.cc:729] T c4bf2462d8b949d6a2f734780fd0d746 P 8a102eff724b4f459bc91203802847fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8a102eff724b4f459bc91203802847fb, State: Initialized, Role: LEARNER
I20240501 08:11:13.625216 26227 consensus_queue.cc:260] T c4bf2462d8b949d6a2f734780fd0d746 P 8a102eff724b4f459bc91203802847fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "8a102eff724b4f459bc91203802847fb" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 35569 } attrs { promote: true } }
I20240501 08:11:13.625705 26227 ts_tablet_manager.cc:1398] T c4bf2462d8b949d6a2f734780fd0d746 P 8a102eff724b4f459bc91203802847fb: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 08:11:13.627635 24379 tablet_copy_service.cc:342] P 213ccbb05e6f44deb4a34d0a7041f7de: Request end of tablet copy session 8a102eff724b4f459bc91203802847fb-c4bf2462d8b949d6a2f734780fd0d746 received from {username='slave'} at 127.21.84.135:56429
I20240501 08:11:13.627797 24379 tablet_copy_service.cc:433] P 213ccbb05e6f44deb4a34d0a7041f7de: ending tablet copy session 8a102eff724b4f459bc91203802847fb-c4bf2462d8b949d6a2f734780fd0d746 on tablet c4bf2462d8b949d6a2f734780fd0d746 with peer 8a102eff724b4f459bc91203802847fb
I20240501 08:11:13.629402 26292 ts_tablet_manager.cc:1398] T b0c941c4d9ba4607b947765c218a1a60 P 962160cac57040678e5cacd4b8b73c4b: Time spent starting tablet: real 0.037s user 0.014s sys 0.006s
I20240501 08:11:13.630605 24248 tablet_copy_service.cc:342] P fb59098726224e7bae383f5c11820b39: Request end of tablet copy session 962160cac57040678e5cacd4b8b73c4b-b0c941c4d9ba4607b947765c218a1a60 received from {username='slave'} at 127.21.84.133:59095
I20240501 08:11:13.630741 24248 tablet_copy_service.cc:433] P fb59098726224e7bae383f5c11820b39: ending tablet copy session 962160cac57040678e5cacd4b8b73c4b-b0c941c4d9ba4607b947765c218a1a60 on tablet b0c941c4d9ba4607b947765c218a1a60 with peer 962160cac57040678e5cacd4b8b73c4b
I20240501 08:11:13.649861 26269 tablet_bootstrap.cc:492] T 0705e5b48e84423fa252da5d62b3008b P 5ce52428c87b46369add4a8afaaef67c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:13.650606 26269 tablet_bootstrap.cc:492] T 0705e5b48e84423fa252da5d62b3008b P 5ce52428c87b46369add4a8afaaef67c: Bootstrap complete.
I20240501 08:11:13.651171 26269 ts_tablet_manager.cc:1367] T 0705e5b48e84423fa252da5d62b3008b P 5ce52428c87b46369add4a8afaaef67c: Time spent bootstrapping tablet: real 0.255s user 0.060s sys 0.013s
I20240501 08:11:13.658186 26269 raft_consensus.cc:348] T 0705e5b48e84423fa252da5d62b3008b P 5ce52428c87b46369add4a8afaaef67c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "5ce52428c87b46369add4a8afaaef67c" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 43389 } attrs { promote: true } }
I20240501 08:11:13.658710 26269 raft_consensus.cc:729] T 0705e5b48e84423fa252da5d62b3008b P 5ce52428c87b46369add4a8afaaef67c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5ce52428c87b46369add4a8afaaef67c, State: Initialized, Role: LEARNER
I20240501 08:11:13.659302 26269 consensus_queue.cc:260] T 0705e5b48e84423fa252da5d62b3008b P 5ce52428c87b46369add4a8afaaef67c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "5ce52428c87b46369add4a8afaaef67c" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 43389 } attrs { promote: true } }
I20240501 08:11:13.659618 26221 tablet_bootstrap.cc:492] T 158f407899b6440cb33d57025ab56873 P f0c82b0e87d84b5eb3b2fd4f3e63ff1b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:13.660223 26221 tablet_bootstrap.cc:492] T 158f407899b6440cb33d57025ab56873 P f0c82b0e87d84b5eb3b2fd4f3e63ff1b: Bootstrap complete.
I20240501 08:11:13.660547 26298 tablet_bootstrap.cc:492] T 19eec94f19284c9e97efdbe449a5238d P 5ce52428c87b46369add4a8afaaef67c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:13.660687 26221 ts_tablet_manager.cc:1367] T 158f407899b6440cb33d57025ab56873 P f0c82b0e87d84b5eb3b2fd4f3e63ff1b: Time spent bootstrapping tablet: real 0.125s user 0.053s sys 0.005s
I20240501 08:11:13.660948 26298 tablet_bootstrap.cc:492] T 19eec94f19284c9e97efdbe449a5238d P 5ce52428c87b46369add4a8afaaef67c: Bootstrap complete.
I20240501 08:11:13.661223 26298 ts_tablet_manager.cc:1367] T 19eec94f19284c9e97efdbe449a5238d P 5ce52428c87b46369add4a8afaaef67c: Time spent bootstrapping tablet: real 0.144s user 0.046s sys 0.009s
I20240501 08:11:13.662249 26298 raft_consensus.cc:348] T 19eec94f19284c9e97efdbe449a5238d P 5ce52428c87b46369add4a8afaaef67c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "5ce52428c87b46369add4a8afaaef67c" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 43389 } attrs { promote: true } }
I20240501 08:11:13.662258 26221 raft_consensus.cc:348] T 158f407899b6440cb33d57025ab56873 P f0c82b0e87d84b5eb3b2fd4f3e63ff1b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 35031 } attrs { promote: true } }
I20240501 08:11:13.662578 26298 raft_consensus.cc:729] T 19eec94f19284c9e97efdbe449a5238d P 5ce52428c87b46369add4a8afaaef67c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5ce52428c87b46369add4a8afaaef67c, State: Initialized, Role: LEARNER
I20240501 08:11:13.662782 26221 raft_consensus.cc:729] T 158f407899b6440cb33d57025ab56873 P f0c82b0e87d84b5eb3b2fd4f3e63ff1b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0c82b0e87d84b5eb3b2fd4f3e63ff1b, State: Initialized, Role: LEARNER
I20240501 08:11:13.662845 26298 consensus_queue.cc:260] T 19eec94f19284c9e97efdbe449a5238d P 5ce52428c87b46369add4a8afaaef67c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "5ce52428c87b46369add4a8afaaef67c" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 43389 } attrs { promote: true } }
I20240501 08:11:13.663203 26221 consensus_queue.cc:260] T 158f407899b6440cb33d57025ab56873 P f0c82b0e87d84b5eb3b2fd4f3e63ff1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 35031 } attrs { promote: true } }
I20240501 08:11:13.664160 26221 ts_tablet_manager.cc:1398] T 158f407899b6440cb33d57025ab56873 P f0c82b0e87d84b5eb3b2fd4f3e63ff1b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:13.665370 24379 tablet_copy_service.cc:342] P 213ccbb05e6f44deb4a34d0a7041f7de: Request end of tablet copy session f0c82b0e87d84b5eb3b2fd4f3e63ff1b-158f407899b6440cb33d57025ab56873 received from {username='slave'} at 127.21.84.131:46057
I20240501 08:11:13.665563 24379 tablet_copy_service.cc:433] P 213ccbb05e6f44deb4a34d0a7041f7de: ending tablet copy session f0c82b0e87d84b5eb3b2fd4f3e63ff1b-158f407899b6440cb33d57025ab56873 on tablet 158f407899b6440cb33d57025ab56873 with peer f0c82b0e87d84b5eb3b2fd4f3e63ff1b
I20240501 08:11:13.680029 26269 ts_tablet_manager.cc:1398] T 0705e5b48e84423fa252da5d62b3008b P 5ce52428c87b46369add4a8afaaef67c: Time spent starting tablet: real 0.029s user 0.016s sys 0.012s
I20240501 08:11:13.680030 26298 ts_tablet_manager.cc:1398] T 19eec94f19284c9e97efdbe449a5238d P 5ce52428c87b46369add4a8afaaef67c: Time spent starting tablet: real 0.019s user 0.009s sys 0.009s
I20240501 08:11:13.681092 24379 tablet_copy_service.cc:342] P 213ccbb05e6f44deb4a34d0a7041f7de: Request end of tablet copy session 5ce52428c87b46369add4a8afaaef67c-0705e5b48e84423fa252da5d62b3008b received from {username='slave'} at 127.21.84.132:37865
I20240501 08:11:13.681254 24379 tablet_copy_service.cc:433] P 213ccbb05e6f44deb4a34d0a7041f7de: ending tablet copy session 5ce52428c87b46369add4a8afaaef67c-0705e5b48e84423fa252da5d62b3008b on tablet 0705e5b48e84423fa252da5d62b3008b with peer 5ce52428c87b46369add4a8afaaef67c
I20240501 08:11:13.681245 24248 tablet_copy_service.cc:342] P fb59098726224e7bae383f5c11820b39: Request end of tablet copy session 5ce52428c87b46369add4a8afaaef67c-19eec94f19284c9e97efdbe449a5238d received from {username='slave'} at 127.21.84.132:43125
I20240501 08:11:13.681365 24248 tablet_copy_service.cc:433] P fb59098726224e7bae383f5c11820b39: ending tablet copy session 5ce52428c87b46369add4a8afaaef67c-19eec94f19284c9e97efdbe449a5238d on tablet 19eec94f19284c9e97efdbe449a5238d with peer 5ce52428c87b46369add4a8afaaef67c
I20240501 08:11:13.686146 26288 tablet_bootstrap.cc:492] T 9b065f9074a74fd9945f22596e7594db P 962160cac57040678e5cacd4b8b73c4b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:13.686524 26288 tablet_bootstrap.cc:492] T 9b065f9074a74fd9945f22596e7594db P 962160cac57040678e5cacd4b8b73c4b: Bootstrap complete.
I20240501 08:11:13.686758 26288 ts_tablet_manager.cc:1367] T 9b065f9074a74fd9945f22596e7594db P 962160cac57040678e5cacd4b8b73c4b: Time spent bootstrapping tablet: real 0.148s user 0.059s sys 0.006s
I20240501 08:11:13.687542 26288 raft_consensus.cc:348] T 9b065f9074a74fd9945f22596e7594db P 962160cac57040678e5cacd4b8b73c4b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "962160cac57040678e5cacd4b8b73c4b" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 37341 } attrs { promote: true } }
I20240501 08:11:13.687729 26288 raft_consensus.cc:729] T 9b065f9074a74fd9945f22596e7594db P 962160cac57040678e5cacd4b8b73c4b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 962160cac57040678e5cacd4b8b73c4b, State: Initialized, Role: LEARNER
I20240501 08:11:13.687904 26288 consensus_queue.cc:260] T 9b065f9074a74fd9945f22596e7594db P 962160cac57040678e5cacd4b8b73c4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "962160cac57040678e5cacd4b8b73c4b" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 37341 } attrs { promote: true } }
I20240501 08:11:13.688369 26288 ts_tablet_manager.cc:1398] T 9b065f9074a74fd9945f22596e7594db P 962160cac57040678e5cacd4b8b73c4b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:11:13.689110 24248 tablet_copy_service.cc:342] P fb59098726224e7bae383f5c11820b39: Request end of tablet copy session 962160cac57040678e5cacd4b8b73c4b-9b065f9074a74fd9945f22596e7594db received from {username='slave'} at 127.21.84.133:59095
I20240501 08:11:13.689224 24248 tablet_copy_service.cc:433] P fb59098726224e7bae383f5c11820b39: ending tablet copy session 962160cac57040678e5cacd4b8b73c4b-9b065f9074a74fd9945f22596e7594db on tablet 9b065f9074a74fd9945f22596e7594db with peer 962160cac57040678e5cacd4b8b73c4b
I20240501 08:11:13.765664 25712 raft_consensus.cc:1203] T 9b065f9074a74fd9945f22596e7594db P 962160cac57040678e5cacd4b8b73c4b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:11:13.773314 25712 raft_consensus.cc:1203] T b0c941c4d9ba4607b947765c218a1a60 P 962160cac57040678e5cacd4b8b73c4b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:11:13.806011 25582 raft_consensus.cc:1203] T 19eec94f19284c9e97efdbe449a5238d P 5ce52428c87b46369add4a8afaaef67c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:11:13.822821 25842 raft_consensus.cc:1203] T 89a8cf2415454cb7b9157703f72913f4 P 1ad8fc587a734cc48f7f49b00fc0c0a1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:11:13.903481 25451 raft_consensus.cc:1203] T 158f407899b6440cb33d57025ab56873 P f0c82b0e87d84b5eb3b2fd4f3e63ff1b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:11:13.949234 26170 raft_consensus.cc:1050] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39: attempting to promote NON_VOTER f0c82b0e87d84b5eb3b2fd4f3e63ff1b to VOTER
I20240501 08:11:13.950294 26170 consensus_queue.cc:237] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 35031 } attrs { promote: false } }
I20240501 08:11:13.951083 26327 raft_consensus.cc:1050] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39: attempting to promote NON_VOTER 8a102eff724b4f459bc91203802847fb to VOTER
I20240501 08:11:13.951714 26327 consensus_queue.cc:237] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "8a102eff724b4f459bc91203802847fb" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 35569 } attrs { promote: false } }
I20240501 08:11:13.952625 25451 raft_consensus.cc:1261] T 476e8260712e414ab298ea94d413044e P f0c82b0e87d84b5eb3b2fd4f3e63ff1b [term 1 LEARNER]: Refusing update from remote peer fb59098726224e7bae383f5c11820b39: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:11:13.953395 26332 consensus_queue.cc:1035] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 35031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:11:13.953811 25972 raft_consensus.cc:1261] T abc7faa250d14cecb417da206b1927a3 P 8a102eff724b4f459bc91203802847fb [term 1 LEARNER]: Refusing update from remote peer fb59098726224e7bae383f5c11820b39: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:11:13.954473 26332 consensus_queue.cc:1035] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a102eff724b4f459bc91203802847fb" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 35569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:11:13.957540 26327 raft_consensus.cc:2932] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f0c82b0e87d84b5eb3b2fd4f3e63ff1b (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 35031 } attrs { promote: false } } }
I20240501 08:11:13.958004 26170 raft_consensus.cc:2932] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a102eff724b4f459bc91203802847fb (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "8a102eff724b4f459bc91203802847fb" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 35569 } attrs { promote: false } } }
I20240501 08:11:13.958436 25451 raft_consensus.cc:2932] T 476e8260712e414ab298ea94d413044e P f0c82b0e87d84b5eb3b2fd4f3e63ff1b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f0c82b0e87d84b5eb3b2fd4f3e63ff1b (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 35031 } attrs { promote: false } } }
I20240501 08:11:13.961730 25972 raft_consensus.cc:2932] T abc7faa250d14cecb417da206b1927a3 P 8a102eff724b4f459bc91203802847fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a102eff724b4f459bc91203802847fb (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "8a102eff724b4f459bc91203802847fb" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 35569 } attrs { promote: false } } }
I20240501 08:11:13.963222 25971 raft_consensus.cc:1203] T c4bf2462d8b949d6a2f734780fd0d746 P 8a102eff724b4f459bc91203802847fb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:11:13.965054 24087 catalog_manager.cc:5484] T 476e8260712e414ab298ea94d413044e P f0c82b0e87d84b5eb3b2fd4f3e63ff1b reported cstate change: config changed from index 4 to 5, f0c82b0e87d84b5eb3b2fd4f3e63ff1b (127.21.84.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fb59098726224e7bae383f5c11820b39" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 35031 } attrs { promote: false } } }
I20240501 08:11:13.970641 24078 catalog_manager.cc:5484] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39 reported cstate change: config changed from index 4 to 5, 8a102eff724b4f459bc91203802847fb (127.21.84.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fb59098726224e7bae383f5c11820b39" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a102eff724b4f459bc91203802847fb" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 35569 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:13.984351 25842 raft_consensus.cc:1203] T 6305a4cfdfc3420594ff185a15723ad1 P 1ad8fc587a734cc48f7f49b00fc0c0a1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:11:14.095162 25582 raft_consensus.cc:1203] T 0705e5b48e84423fa252da5d62b3008b P 5ce52428c87b46369add4a8afaaef67c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:11:14.163223 26327 raft_consensus.cc:1050] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39: attempting to promote NON_VOTER 962160cac57040678e5cacd4b8b73c4b to VOTER
I20240501 08:11:14.163971 26327 consensus_queue.cc:237] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "962160cac57040678e5cacd4b8b73c4b" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 37341 } attrs { promote: false } }
I20240501 08:11:14.166337 25712 raft_consensus.cc:1261] T b0c941c4d9ba4607b947765c218a1a60 P 962160cac57040678e5cacd4b8b73c4b [term 1 LEARNER]: Refusing update from remote peer fb59098726224e7bae383f5c11820b39: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:11:14.167043 26329 consensus_queue.cc:1035] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "962160cac57040678e5cacd4b8b73c4b" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 37341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:11:14.171526 26327 raft_consensus.cc:2932] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 962160cac57040678e5cacd4b8b73c4b (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "962160cac57040678e5cacd4b8b73c4b" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 37341 } attrs { promote: false } } }
I20240501 08:11:14.172888 25712 raft_consensus.cc:2932] T b0c941c4d9ba4607b947765c218a1a60 P 962160cac57040678e5cacd4b8b73c4b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 962160cac57040678e5cacd4b8b73c4b (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "962160cac57040678e5cacd4b8b73c4b" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 37341 } attrs { promote: false } } }
I20240501 08:11:14.174160 24228 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "476e8260712e414ab298ea94d413044e"
dest_uuid: "fb59098726224e7bae383f5c11820b39"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46022
I20240501 08:11:14.174439 24228 raft_consensus.cc:595] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:14.177057 24078 catalog_manager.cc:5484] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39 reported cstate change: config changed from index 2 to 3, 962160cac57040678e5cacd4b8b73c4b (127.21.84.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fb59098726224e7bae383f5c11820b39" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "962160cac57040678e5cacd4b8b73c4b" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 37341 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:14.222790 24228 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "abc7faa250d14cecb417da206b1927a3"
dest_uuid: "fb59098726224e7bae383f5c11820b39"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46048
I20240501 08:11:14.223029 24228 raft_consensus.cc:595] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:14.262017 26392 raft_consensus.cc:1050] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de: attempting to promote NON_VOTER 1ad8fc587a734cc48f7f49b00fc0c0a1 to VOTER
I20240501 08:11:14.262842 26392 consensus_queue.cc:237] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 45565 } attrs { promote: false } }
I20240501 08:11:14.264732 25842 raft_consensus.cc:1261] T 89a8cf2415454cb7b9157703f72913f4 P 1ad8fc587a734cc48f7f49b00fc0c0a1 [term 1 LEARNER]: Refusing update from remote peer 213ccbb05e6f44deb4a34d0a7041f7de: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:11:14.265370 26106 consensus_queue.cc:1035] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 45565 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:11:14.268767 26392 raft_consensus.cc:2932] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1ad8fc587a734cc48f7f49b00fc0c0a1 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 45565 } attrs { promote: false } } }
I20240501 08:11:14.269472 25842 raft_consensus.cc:2932] T 89a8cf2415454cb7b9157703f72913f4 P 1ad8fc587a734cc48f7f49b00fc0c0a1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1ad8fc587a734cc48f7f49b00fc0c0a1 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 45565 } attrs { promote: false } } }
I20240501 08:11:14.273451 24078 catalog_manager.cc:5484] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de reported cstate change: config changed from index 4 to 5, 1ad8fc587a734cc48f7f49b00fc0c0a1 (127.21.84.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 45565 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:14.334120 26329 raft_consensus.cc:1050] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39: attempting to promote NON_VOTER 962160cac57040678e5cacd4b8b73c4b to VOTER
I20240501 08:11:14.334748 26329 consensus_queue.cc:237] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "962160cac57040678e5cacd4b8b73c4b" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 37341 } attrs { promote: false } }
I20240501 08:11:14.336678 25712 raft_consensus.cc:1261] T 9b065f9074a74fd9945f22596e7594db P 962160cac57040678e5cacd4b8b73c4b [term 1 LEARNER]: Refusing update from remote peer fb59098726224e7bae383f5c11820b39: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:11:14.337260 26327 consensus_queue.cc:1035] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "962160cac57040678e5cacd4b8b73c4b" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 37341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:11:14.339699 26329 raft_consensus.cc:2932] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 962160cac57040678e5cacd4b8b73c4b (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "962160cac57040678e5cacd4b8b73c4b" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 37341 } attrs { promote: false } } }
I20240501 08:11:14.340358 25712 raft_consensus.cc:2932] T 9b065f9074a74fd9945f22596e7594db P 962160cac57040678e5cacd4b8b73c4b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 962160cac57040678e5cacd4b8b73c4b (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "962160cac57040678e5cacd4b8b73c4b" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 37341 } attrs { promote: false } } }
I20240501 08:11:14.344863 24080 catalog_manager.cc:5484] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39 reported cstate change: config changed from index 4 to 5, 962160cac57040678e5cacd4b8b73c4b (127.21.84.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fb59098726224e7bae383f5c11820b39" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "962160cac57040678e5cacd4b8b73c4b" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 37341 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:14.373548 26329 raft_consensus.cc:1050] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39: attempting to promote NON_VOTER 5ce52428c87b46369add4a8afaaef67c to VOTER
I20240501 08:11:14.374121 26329 consensus_queue.cc:237] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "5ce52428c87b46369add4a8afaaef67c" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 43389 } attrs { promote: false } }
I20240501 08:11:14.375977 25582 raft_consensus.cc:1261] T 19eec94f19284c9e97efdbe449a5238d P 5ce52428c87b46369add4a8afaaef67c [term 1 LEARNER]: Refusing update from remote peer fb59098726224e7bae383f5c11820b39: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:11:14.376616 26329 consensus_queue.cc:1035] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce52428c87b46369add4a8afaaef67c" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 43389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:11:14.380014 26332 raft_consensus.cc:2932] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5ce52428c87b46369add4a8afaaef67c (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "5ce52428c87b46369add4a8afaaef67c" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 43389 } attrs { promote: false } } }
I20240501 08:11:14.380748 25582 raft_consensus.cc:2932] T 19eec94f19284c9e97efdbe449a5238d P 5ce52428c87b46369add4a8afaaef67c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5ce52428c87b46369add4a8afaaef67c (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "5ce52428c87b46369add4a8afaaef67c" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 43389 } attrs { promote: false } } }
I20240501 08:11:14.384454 24078 catalog_manager.cc:5484] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39 reported cstate change: config changed from index 4 to 5, 5ce52428c87b46369add4a8afaaef67c (127.21.84.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fb59098726224e7bae383f5c11820b39" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ce52428c87b46369add4a8afaaef67c" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 43389 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:14.387140 26106 raft_consensus.cc:1050] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de: attempting to promote NON_VOTER 1ad8fc587a734cc48f7f49b00fc0c0a1 to VOTER
I20240501 08:11:14.387842 26106 consensus_queue.cc:237] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 45565 } attrs { promote: false } }
I20240501 08:11:14.389112 26392 raft_consensus.cc:1050] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de: attempting to promote NON_VOTER 8a102eff724b4f459bc91203802847fb to VOTER
I20240501 08:11:14.389770 26392 consensus_queue.cc:237] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "8a102eff724b4f459bc91203802847fb" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 35569 } attrs { promote: false } }
I20240501 08:11:14.391773 25842 raft_consensus.cc:1261] T 6305a4cfdfc3420594ff185a15723ad1 P 1ad8fc587a734cc48f7f49b00fc0c0a1 [term 1 LEARNER]: Refusing update from remote peer 213ccbb05e6f44deb4a34d0a7041f7de: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:11:14.391796 25972 raft_consensus.cc:1261] T c4bf2462d8b949d6a2f734780fd0d746 P 8a102eff724b4f459bc91203802847fb [term 1 LEARNER]: Refusing update from remote peer 213ccbb05e6f44deb4a34d0a7041f7de: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:11:14.392379 26392 consensus_queue.cc:1035] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 45565 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:11:14.392302 26344 consensus_queue.cc:1035] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a102eff724b4f459bc91203802847fb" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 35569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:11:14.393252 26329 raft_consensus.cc:979] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39: : Instructing follower f0c82b0e87d84b5eb3b2fd4f3e63ff1b to start an election
I20240501 08:11:14.393404 26329 raft_consensus.cc:1067] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Signalling peer f0c82b0e87d84b5eb3b2fd4f3e63ff1b to start an election
I20240501 08:11:14.394418 25451 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "476e8260712e414ab298ea94d413044e"
dest_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b"
from {username='slave'} at 127.21.84.129:42841
I20240501 08:11:14.394774 25451 raft_consensus.cc:482] T 476e8260712e414ab298ea94d413044e P f0c82b0e87d84b5eb3b2fd4f3e63ff1b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:11:14.394982 25451 raft_consensus.cc:3037] T 476e8260712e414ab298ea94d413044e P f0c82b0e87d84b5eb3b2fd4f3e63ff1b [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:14.395711 26392 raft_consensus.cc:2932] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a102eff724b4f459bc91203802847fb (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "8a102eff724b4f459bc91203802847fb" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 35569 } attrs { promote: false } } }
I20240501 08:11:14.396512 25972 raft_consensus.cc:2932] T c4bf2462d8b949d6a2f734780fd0d746 P 8a102eff724b4f459bc91203802847fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a102eff724b4f459bc91203802847fb (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "8a102eff724b4f459bc91203802847fb" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 35569 } attrs { promote: false } } }
I20240501 08:11:14.397622 25451 raft_consensus.cc:504] T 476e8260712e414ab298ea94d413044e P f0c82b0e87d84b5eb3b2fd4f3e63ff1b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 35031 } attrs { promote: false } }
I20240501 08:11:14.398857 26344 raft_consensus.cc:2932] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1ad8fc587a734cc48f7f49b00fc0c0a1 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 45565 } attrs { promote: false } } }
I20240501 08:11:14.400197 24228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "476e8260712e414ab298ea94d413044e" candidate_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "fb59098726224e7bae383f5c11820b39"
I20240501 08:11:14.400506 24228 raft_consensus.cc:3032] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:11:14.400593 24228 raft_consensus.cc:729] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb59098726224e7bae383f5c11820b39, State: Running, Role: LEADER
I20240501 08:11:14.400929 24228 consensus_queue.cc:260] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 35031 } attrs { promote: false } }
I20240501 08:11:14.401196 24228 raft_consensus.cc:3037] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:14.402917 24078 catalog_manager.cc:5484] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de reported cstate change: config changed from index 4 to 5, 8a102eff724b4f459bc91203802847fb (127.21.84.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a102eff724b4f459bc91203802847fb" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 35569 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:14.404372 24228 raft_consensus.cc:2454] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0c82b0e87d84b5eb3b2fd4f3e63ff1b in term 2.
I20240501 08:11:14.405535 25386 leader_election.cc:302] T 476e8260712e414ab298ea94d413044e P f0c82b0e87d84b5eb3b2fd4f3e63ff1b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: f0c82b0e87d84b5eb3b2fd4f3e63ff1b, fb59098726224e7bae383f5c11820b39; no voters:
I20240501 08:11:14.406037 26255 raft_consensus.cc:2781] T 476e8260712e414ab298ea94d413044e P f0c82b0e87d84b5eb3b2fd4f3e63ff1b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:11:14.406560 25451 leader_election.cc:288] T 476e8260712e414ab298ea94d413044e P f0c82b0e87d84b5eb3b2fd4f3e63ff1b [CANDIDATE]: Term 2 election: Requested vote from peers fb59098726224e7bae383f5c11820b39 (127.21.84.129:44759)
I20240501 08:11:14.406649 25842 raft_consensus.cc:2932] T 6305a4cfdfc3420594ff185a15723ad1 P 1ad8fc587a734cc48f7f49b00fc0c0a1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1ad8fc587a734cc48f7f49b00fc0c0a1 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 45565 } attrs { promote: false } } }
I20240501 08:11:14.409776 24080 catalog_manager.cc:5484] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de reported cstate change: config changed from index 4 to 5, 1ad8fc587a734cc48f7f49b00fc0c0a1 (127.21.84.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 45565 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:14.410358 26392 raft_consensus.cc:1050] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de: attempting to promote NON_VOTER f0c82b0e87d84b5eb3b2fd4f3e63ff1b to VOTER
I20240501 08:11:14.411223 26392 consensus_queue.cc:237] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 35031 } attrs { promote: false } }
I20240501 08:11:14.413336 25451 raft_consensus.cc:1261] T 158f407899b6440cb33d57025ab56873 P f0c82b0e87d84b5eb3b2fd4f3e63ff1b [term 1 LEARNER]: Refusing update from remote peer 213ccbb05e6f44deb4a34d0a7041f7de: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:11:14.413864 26106 consensus_queue.cc:1035] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 35031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:11:14.414319 26255 raft_consensus.cc:686] T 476e8260712e414ab298ea94d413044e P f0c82b0e87d84b5eb3b2fd4f3e63ff1b [term 2 LEADER]: Becoming Leader. State: Replica: f0c82b0e87d84b5eb3b2fd4f3e63ff1b, State: Running, Role: LEADER
I20240501 08:11:14.414860 26255 consensus_queue.cc:237] T 476e8260712e414ab298ea94d413044e P f0c82b0e87d84b5eb3b2fd4f3e63ff1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 35031 } attrs { promote: false } }
I20240501 08:11:14.416808 26392 raft_consensus.cc:2932] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f0c82b0e87d84b5eb3b2fd4f3e63ff1b (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 35031 } attrs { promote: false } } }
I20240501 08:11:14.417609 25451 raft_consensus.cc:2932] T 158f407899b6440cb33d57025ab56873 P f0c82b0e87d84b5eb3b2fd4f3e63ff1b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f0c82b0e87d84b5eb3b2fd4f3e63ff1b (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 35031 } attrs { promote: false } } }
I20240501 08:11:14.418357 24078 catalog_manager.cc:5484] T 476e8260712e414ab298ea94d413044e P f0c82b0e87d84b5eb3b2fd4f3e63ff1b reported cstate change: term changed from 1 to 2, leader changed from fb59098726224e7bae383f5c11820b39 (127.21.84.129) to f0c82b0e87d84b5eb3b2fd4f3e63ff1b (127.21.84.131). New cstate: current_term: 2 leader_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 35031 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:14.422219 24080 catalog_manager.cc:5484] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de reported cstate change: config changed from index 4 to 5, f0c82b0e87d84b5eb3b2fd4f3e63ff1b (127.21.84.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 35031 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 08:11:14.431262 24071 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 476e8260712e414ab298ea94d413044e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader f0c82b0e87d84b5eb3b2fd4f3e63ff1b (127.21.84.131:35031) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240501 08:11:14.450320 24228 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b0c941c4d9ba4607b947765c218a1a60"
dest_uuid: "fb59098726224e7bae383f5c11820b39"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46066
I20240501 08:11:14.450549 24228 raft_consensus.cc:595] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:14.467947 24359 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6305a4cfdfc3420594ff185a15723ad1"
dest_uuid: "213ccbb05e6f44deb4a34d0a7041f7de"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39398
I20240501 08:11:14.468225 24359 raft_consensus.cc:595] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:14.472999 26332 raft_consensus.cc:979] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39: : Instructing follower 8a102eff724b4f459bc91203802847fb to start an election
I20240501 08:11:14.473148 26332 raft_consensus.cc:1067] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Signalling peer 8a102eff724b4f459bc91203802847fb to start an election
I20240501 08:11:14.473888 25972 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "abc7faa250d14cecb417da206b1927a3"
dest_uuid: "8a102eff724b4f459bc91203802847fb"
from {username='slave'} at 127.21.84.129:54197
I20240501 08:11:14.474192 25972 raft_consensus.cc:482] T abc7faa250d14cecb417da206b1927a3 P 8a102eff724b4f459bc91203802847fb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:11:14.474375 25972 raft_consensus.cc:3037] T abc7faa250d14cecb417da206b1927a3 P 8a102eff724b4f459bc91203802847fb [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:14.476384 25972 raft_consensus.cc:504] T abc7faa250d14cecb417da206b1927a3 P 8a102eff724b4f459bc91203802847fb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "8a102eff724b4f459bc91203802847fb" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 35569 } attrs { promote: false } }
I20240501 08:11:14.477813 25972 leader_election.cc:288] T abc7faa250d14cecb417da206b1927a3 P 8a102eff724b4f459bc91203802847fb [CANDIDATE]: Term 2 election: Requested vote from peers fb59098726224e7bae383f5c11820b39 (127.21.84.129:44759)
I20240501 08:11:14.478361 24228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abc7faa250d14cecb417da206b1927a3" candidate_uuid: "8a102eff724b4f459bc91203802847fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "fb59098726224e7bae383f5c11820b39"
I20240501 08:11:14.478607 24228 raft_consensus.cc:3032] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:11:14.478688 24228 raft_consensus.cc:729] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb59098726224e7bae383f5c11820b39, State: Running, Role: LEADER
I20240501 08:11:14.478987 24228 consensus_queue.cc:260] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "8a102eff724b4f459bc91203802847fb" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 35569 } attrs { promote: false } }
I20240501 08:11:14.479259 24228 raft_consensus.cc:3037] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:14.481251 24228 raft_consensus.cc:2454] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a102eff724b4f459bc91203802847fb in term 2.
I20240501 08:11:14.481876 25907 leader_election.cc:302] T abc7faa250d14cecb417da206b1927a3 P 8a102eff724b4f459bc91203802847fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8a102eff724b4f459bc91203802847fb, fb59098726224e7bae383f5c11820b39; no voters:
I20240501 08:11:14.482326 26263 raft_consensus.cc:2781] T abc7faa250d14cecb417da206b1927a3 P 8a102eff724b4f459bc91203802847fb [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:11:14.482764 26263 raft_consensus.cc:686] T abc7faa250d14cecb417da206b1927a3 P 8a102eff724b4f459bc91203802847fb [term 2 LEADER]: Becoming Leader. State: Replica: 8a102eff724b4f459bc91203802847fb, State: Running, Role: LEADER
I20240501 08:11:14.483258 26263 consensus_queue.cc:237] T abc7faa250d14cecb417da206b1927a3 P 8a102eff724b4f459bc91203802847fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "8a102eff724b4f459bc91203802847fb" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 35569 } attrs { promote: false } }
I20240501 08:11:14.486649 24078 catalog_manager.cc:5484] T abc7faa250d14cecb417da206b1927a3 P 8a102eff724b4f459bc91203802847fb reported cstate change: term changed from 1 to 2, leader changed from fb59098726224e7bae383f5c11820b39 (127.21.84.129) to 8a102eff724b4f459bc91203802847fb (127.21.84.135). New cstate: current_term: 2 leader_uuid: "8a102eff724b4f459bc91203802847fb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a102eff724b4f459bc91203802847fb" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 35569 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:14.488413 24359 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "158f407899b6440cb33d57025ab56873"
dest_uuid: "213ccbb05e6f44deb4a34d0a7041f7de"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39416
I20240501 08:11:14.488644 24359 raft_consensus.cc:595] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:14.507486 24359 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c4bf2462d8b949d6a2f734780fd0d746"
dest_uuid: "213ccbb05e6f44deb4a34d0a7041f7de"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39432
I20240501 08:11:14.507721 24359 raft_consensus.cc:595] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:14.544234 24359 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "89a8cf2415454cb7b9157703f72913f4"
dest_uuid: "213ccbb05e6f44deb4a34d0a7041f7de"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39466
I20240501 08:11:14.544487 24359 raft_consensus.cc:595] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:14.561261 24228 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9b065f9074a74fd9945f22596e7594db"
dest_uuid: "fb59098726224e7bae383f5c11820b39"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46076
I20240501 08:11:14.561479 24228 raft_consensus.cc:595] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:14.575429 26392 raft_consensus.cc:1050] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de: attempting to promote NON_VOTER 5ce52428c87b46369add4a8afaaef67c to VOTER
I20240501 08:11:14.576121 26392 consensus_queue.cc:237] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "5ce52428c87b46369add4a8afaaef67c" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 43389 } attrs { promote: false } }
I20240501 08:11:14.578708 25582 raft_consensus.cc:1261] T 0705e5b48e84423fa252da5d62b3008b P 5ce52428c87b46369add4a8afaaef67c [term 1 LEARNER]: Refusing update from remote peer 213ccbb05e6f44deb4a34d0a7041f7de: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:11:14.579270 26106 consensus_queue.cc:1035] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce52428c87b46369add4a8afaaef67c" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 43389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:11:14.582686 26392 raft_consensus.cc:2932] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5ce52428c87b46369add4a8afaaef67c (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "5ce52428c87b46369add4a8afaaef67c" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 43389 } attrs { promote: false } } }
I20240501 08:11:14.584213 25582 raft_consensus.cc:2932] T 0705e5b48e84423fa252da5d62b3008b P 5ce52428c87b46369add4a8afaaef67c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5ce52428c87b46369add4a8afaaef67c (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "5ce52428c87b46369add4a8afaaef67c" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 43389 } attrs { promote: false } } }
I20240501 08:11:14.589087 24078 catalog_manager.cc:5484] T 0705e5b48e84423fa252da5d62b3008b P 5ce52428c87b46369add4a8afaaef67c reported cstate change: config changed from index 4 to 5, 5ce52428c87b46369add4a8afaaef67c (127.21.84.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "5ce52428c87b46369add4a8afaaef67c" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 43389 } attrs { promote: false } } }
I20240501 08:11:14.596978 24228 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "19eec94f19284c9e97efdbe449a5238d"
dest_uuid: "fb59098726224e7bae383f5c11820b39"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46090
I20240501 08:11:14.597214 24228 raft_consensus.cc:595] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:14.611946 26329 raft_consensus.cc:979] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39: : Instructing follower 962160cac57040678e5cacd4b8b73c4b to start an election
I20240501 08:11:14.612145 26329 raft_consensus.cc:1067] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Signalling peer 962160cac57040678e5cacd4b8b73c4b to start an election
I20240501 08:11:14.612880 25712 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b0c941c4d9ba4607b947765c218a1a60"
dest_uuid: "962160cac57040678e5cacd4b8b73c4b"
from {username='slave'} at 127.21.84.129:33263
I20240501 08:11:14.613174 25712 raft_consensus.cc:482] T b0c941c4d9ba4607b947765c218a1a60 P 962160cac57040678e5cacd4b8b73c4b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:11:14.613364 25712 raft_consensus.cc:3037] T b0c941c4d9ba4607b947765c218a1a60 P 962160cac57040678e5cacd4b8b73c4b [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:14.615516 25712 raft_consensus.cc:504] T b0c941c4d9ba4607b947765c218a1a60 P 962160cac57040678e5cacd4b8b73c4b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "962160cac57040678e5cacd4b8b73c4b" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 37341 } attrs { promote: false } }
I20240501 08:11:14.616925 25712 leader_election.cc:288] T b0c941c4d9ba4607b947765c218a1a60 P 962160cac57040678e5cacd4b8b73c4b [CANDIDATE]: Term 2 election: Requested vote from peers fb59098726224e7bae383f5c11820b39 (127.21.84.129:44759)
I20240501 08:11:14.617511 24228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0c941c4d9ba4607b947765c218a1a60" candidate_uuid: "962160cac57040678e5cacd4b8b73c4b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fb59098726224e7bae383f5c11820b39"
I20240501 08:11:14.617748 24228 raft_consensus.cc:3032] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:11:14.617831 24228 raft_consensus.cc:729] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb59098726224e7bae383f5c11820b39, State: Running, Role: LEADER
I20240501 08:11:14.618126 24228 consensus_queue.cc:260] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "962160cac57040678e5cacd4b8b73c4b" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 37341 } attrs { promote: false } }
I20240501 08:11:14.618458 24228 raft_consensus.cc:3037] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:14.620460 24228 raft_consensus.cc:2454] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 962160cac57040678e5cacd4b8b73c4b in term 2.
I20240501 08:11:14.620978 25647 leader_election.cc:302] T b0c941c4d9ba4607b947765c218a1a60 P 962160cac57040678e5cacd4b8b73c4b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 962160cac57040678e5cacd4b8b73c4b, fb59098726224e7bae383f5c11820b39; no voters:
I20240501 08:11:14.621352 26343 raft_consensus.cc:2781] T b0c941c4d9ba4607b947765c218a1a60 P 962160cac57040678e5cacd4b8b73c4b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:11:14.621752 26343 raft_consensus.cc:686] T b0c941c4d9ba4607b947765c218a1a60 P 962160cac57040678e5cacd4b8b73c4b [term 2 LEADER]: Becoming Leader. State: Replica: 962160cac57040678e5cacd4b8b73c4b, State: Running, Role: LEADER
I20240501 08:11:14.622215 26343 consensus_queue.cc:237] T b0c941c4d9ba4607b947765c218a1a60 P 962160cac57040678e5cacd4b8b73c4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "962160cac57040678e5cacd4b8b73c4b" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 37341 } attrs { promote: false } }
I20240501 08:11:14.625630 24080 catalog_manager.cc:5484] T b0c941c4d9ba4607b947765c218a1a60 P 962160cac57040678e5cacd4b8b73c4b reported cstate change: term changed from 1 to 2, leader changed from fb59098726224e7bae383f5c11820b39 (127.21.84.129) to 962160cac57040678e5cacd4b8b73c4b (127.21.84.133). New cstate: current_term: 2 leader_uuid: "962160cac57040678e5cacd4b8b73c4b" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "962160cac57040678e5cacd4b8b73c4b" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 37341 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:14.752271 26332 raft_consensus.cc:979] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39: : Instructing follower 962160cac57040678e5cacd4b8b73c4b to start an election
I20240501 08:11:14.752480 26332 raft_consensus.cc:1067] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Signalling peer 962160cac57040678e5cacd4b8b73c4b to start an election
I20240501 08:11:14.753098 25712 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9b065f9074a74fd9945f22596e7594db"
dest_uuid: "962160cac57040678e5cacd4b8b73c4b"
from {username='slave'} at 127.21.84.129:33263
I20240501 08:11:14.753321 25712 raft_consensus.cc:482] T 9b065f9074a74fd9945f22596e7594db P 962160cac57040678e5cacd4b8b73c4b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:11:14.753446 25712 raft_consensus.cc:3037] T 9b065f9074a74fd9945f22596e7594db P 962160cac57040678e5cacd4b8b73c4b [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:14.755486 25712 raft_consensus.cc:504] T 9b065f9074a74fd9945f22596e7594db P 962160cac57040678e5cacd4b8b73c4b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "962160cac57040678e5cacd4b8b73c4b" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 37341 } attrs { promote: false } }
I20240501 08:11:14.755951 25712 leader_election.cc:288] T 9b065f9074a74fd9945f22596e7594db P 962160cac57040678e5cacd4b8b73c4b [CANDIDATE]: Term 2 election: Requested vote from peers fb59098726224e7bae383f5c11820b39 (127.21.84.129:44759)
I20240501 08:11:14.756611 24228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b065f9074a74fd9945f22596e7594db" candidate_uuid: "962160cac57040678e5cacd4b8b73c4b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "fb59098726224e7bae383f5c11820b39"
I20240501 08:11:14.756822 24228 raft_consensus.cc:3032] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:11:14.756906 24228 raft_consensus.cc:729] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb59098726224e7bae383f5c11820b39, State: Running, Role: LEADER
I20240501 08:11:14.757126 24228 consensus_queue.cc:260] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "962160cac57040678e5cacd4b8b73c4b" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 37341 } attrs { promote: false } }
I20240501 08:11:14.757354 24228 raft_consensus.cc:3037] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:14.759819 24228 raft_consensus.cc:2454] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 962160cac57040678e5cacd4b8b73c4b in term 2.
I20240501 08:11:14.760252 25647 leader_election.cc:302] T 9b065f9074a74fd9945f22596e7594db P 962160cac57040678e5cacd4b8b73c4b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 962160cac57040678e5cacd4b8b73c4b, fb59098726224e7bae383f5c11820b39; no voters:
I20240501 08:11:14.760560 26343 raft_consensus.cc:2781] T 9b065f9074a74fd9945f22596e7594db P 962160cac57040678e5cacd4b8b73c4b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:11:14.760730 26343 raft_consensus.cc:686] T 9b065f9074a74fd9945f22596e7594db P 962160cac57040678e5cacd4b8b73c4b [term 2 LEADER]: Becoming Leader. State: Replica: 962160cac57040678e5cacd4b8b73c4b, State: Running, Role: LEADER
I20240501 08:11:14.760968 26343 consensus_queue.cc:237] T 9b065f9074a74fd9945f22596e7594db P 962160cac57040678e5cacd4b8b73c4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "962160cac57040678e5cacd4b8b73c4b" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 37341 } attrs { promote: false } }
I20240501 08:11:14.763044 24080 catalog_manager.cc:5484] T 9b065f9074a74fd9945f22596e7594db P 962160cac57040678e5cacd4b8b73c4b reported cstate change: term changed from 1 to 2, leader changed from fb59098726224e7bae383f5c11820b39 (127.21.84.129) to 962160cac57040678e5cacd4b8b73c4b (127.21.84.133). New cstate: current_term: 2 leader_uuid: "962160cac57040678e5cacd4b8b73c4b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "962160cac57040678e5cacd4b8b73c4b" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 37341 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:14.803790 26344 raft_consensus.cc:979] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de: : Instructing follower 1ad8fc587a734cc48f7f49b00fc0c0a1 to start an election
I20240501 08:11:14.804003 26344 raft_consensus.cc:1067] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Signalling peer 1ad8fc587a734cc48f7f49b00fc0c0a1 to start an election
I20240501 08:11:14.804992 25842 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "89a8cf2415454cb7b9157703f72913f4"
dest_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1"
from {username='slave'} at 127.21.84.130:50217
I20240501 08:11:14.805321 25842 raft_consensus.cc:482] T 89a8cf2415454cb7b9157703f72913f4 P 1ad8fc587a734cc48f7f49b00fc0c0a1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:11:14.805538 25842 raft_consensus.cc:3037] T 89a8cf2415454cb7b9157703f72913f4 P 1ad8fc587a734cc48f7f49b00fc0c0a1 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:14.808267 25842 raft_consensus.cc:504] T 89a8cf2415454cb7b9157703f72913f4 P 1ad8fc587a734cc48f7f49b00fc0c0a1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 45565 } attrs { promote: false } }
I20240501 08:11:14.809826 25842 leader_election.cc:288] T 89a8cf2415454cb7b9157703f72913f4 P 1ad8fc587a734cc48f7f49b00fc0c0a1 [CANDIDATE]: Term 2 election: Requested vote from peers 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130:37857)
I20240501 08:11:14.810487 24359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a8cf2415454cb7b9157703f72913f4" candidate_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "213ccbb05e6f44deb4a34d0a7041f7de"
I20240501 08:11:14.810875 24359 raft_consensus.cc:3032] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:11:14.810972 24359 raft_consensus.cc:729] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 213ccbb05e6f44deb4a34d0a7041f7de, State: Running, Role: LEADER
I20240501 08:11:14.811286 24359 consensus_queue.cc:260] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 45565 } attrs { promote: false } }
I20240501 08:11:14.811687 24359 raft_consensus.cc:3037] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:14.815534 24359 raft_consensus.cc:2454] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ad8fc587a734cc48f7f49b00fc0c0a1 in term 2.
I20240501 08:11:14.818323 25778 leader_election.cc:302] T 89a8cf2415454cb7b9157703f72913f4 P 1ad8fc587a734cc48f7f49b00fc0c0a1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1ad8fc587a734cc48f7f49b00fc0c0a1, 213ccbb05e6f44deb4a34d0a7041f7de; no voters:
I20240501 08:11:14.818800 26323 raft_consensus.cc:2781] T 89a8cf2415454cb7b9157703f72913f4 P 1ad8fc587a734cc48f7f49b00fc0c0a1 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:11:14.819231 26323 raft_consensus.cc:686] T 89a8cf2415454cb7b9157703f72913f4 P 1ad8fc587a734cc48f7f49b00fc0c0a1 [term 2 LEADER]: Becoming Leader. State: Replica: 1ad8fc587a734cc48f7f49b00fc0c0a1, State: Running, Role: LEADER
I20240501 08:11:14.819737 26323 consensus_queue.cc:237] T 89a8cf2415454cb7b9157703f72913f4 P 1ad8fc587a734cc48f7f49b00fc0c0a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 45565 } attrs { promote: false } }
I20240501 08:11:14.822934 24080 catalog_manager.cc:5484] T 89a8cf2415454cb7b9157703f72913f4 P 1ad8fc587a734cc48f7f49b00fc0c0a1 reported cstate change: term changed from 1 to 2, leader changed from 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130) to 1ad8fc587a734cc48f7f49b00fc0c0a1 (127.21.84.134). New cstate: current_term: 2 leader_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 45565 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:14.834972 24359 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6305a4cfdfc3420594ff185a15723ad1"
dest_uuid: "213ccbb05e6f44deb4a34d0a7041f7de"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39476
I20240501 08:11:14.835218 24359 raft_consensus.cc:595] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:14.843413 26332 raft_consensus.cc:979] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39: : Instructing follower 5ce52428c87b46369add4a8afaaef67c to start an election
I20240501 08:11:14.843662 26332 raft_consensus.cc:1067] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Signalling peer 5ce52428c87b46369add4a8afaaef67c to start an election
I20240501 08:11:14.844952 25582 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "19eec94f19284c9e97efdbe449a5238d"
dest_uuid: "5ce52428c87b46369add4a8afaaef67c"
from {username='slave'} at 127.21.84.129:46671
I20240501 08:11:14.845281 25582 raft_consensus.cc:482] T 19eec94f19284c9e97efdbe449a5238d P 5ce52428c87b46369add4a8afaaef67c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:11:14.845486 25582 raft_consensus.cc:3037] T 19eec94f19284c9e97efdbe449a5238d P 5ce52428c87b46369add4a8afaaef67c [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:14.848603 25582 raft_consensus.cc:504] T 19eec94f19284c9e97efdbe449a5238d P 5ce52428c87b46369add4a8afaaef67c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "5ce52428c87b46369add4a8afaaef67c" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 43389 } attrs { promote: false } }
I20240501 08:11:14.850535 25582 leader_election.cc:288] T 19eec94f19284c9e97efdbe449a5238d P 5ce52428c87b46369add4a8afaaef67c [CANDIDATE]: Term 2 election: Requested vote from peers fb59098726224e7bae383f5c11820b39 (127.21.84.129:44759)
I20240501 08:11:14.851194 24228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19eec94f19284c9e97efdbe449a5238d" candidate_uuid: "5ce52428c87b46369add4a8afaaef67c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "fb59098726224e7bae383f5c11820b39"
I20240501 08:11:14.851419 24228 raft_consensus.cc:3032] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:11:14.851506 24228 raft_consensus.cc:729] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb59098726224e7bae383f5c11820b39, State: Running, Role: LEADER
I20240501 08:11:14.851820 24228 consensus_queue.cc:260] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "5ce52428c87b46369add4a8afaaef67c" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 43389 } attrs { promote: false } }
I20240501 08:11:14.852134 24228 raft_consensus.cc:3037] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:14.855182 24228 raft_consensus.cc:2454] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce52428c87b46369add4a8afaaef67c in term 2.
I20240501 08:11:14.855963 25517 leader_election.cc:302] T 19eec94f19284c9e97efdbe449a5238d P 5ce52428c87b46369add4a8afaaef67c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5ce52428c87b46369add4a8afaaef67c, fb59098726224e7bae383f5c11820b39; no voters:
I20240501 08:11:14.856551 26345 raft_consensus.cc:2781] T 19eec94f19284c9e97efdbe449a5238d P 5ce52428c87b46369add4a8afaaef67c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:11:14.870265 26345 raft_consensus.cc:686] T 19eec94f19284c9e97efdbe449a5238d P 5ce52428c87b46369add4a8afaaef67c [term 2 LEADER]: Becoming Leader. State: Replica: 5ce52428c87b46369add4a8afaaef67c, State: Running, Role: LEADER
I20240501 08:11:14.871031 26345 consensus_queue.cc:237] T 19eec94f19284c9e97efdbe449a5238d P 5ce52428c87b46369add4a8afaaef67c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "5ce52428c87b46369add4a8afaaef67c" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 43389 } attrs { promote: false } }
I20240501 08:11:14.872725 26478 raft_consensus.cc:979] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de: : Instructing follower f0c82b0e87d84b5eb3b2fd4f3e63ff1b to start an election
I20240501 08:11:14.872896 26478 raft_consensus.cc:1067] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Signalling peer f0c82b0e87d84b5eb3b2fd4f3e63ff1b to start an election
I20240501 08:11:14.873773 25451 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "158f407899b6440cb33d57025ab56873"
dest_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b"
from {username='slave'} at 127.21.84.130:55487
I20240501 08:11:14.874034 25451 raft_consensus.cc:482] T 158f407899b6440cb33d57025ab56873 P f0c82b0e87d84b5eb3b2fd4f3e63ff1b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:11:14.874152 25451 raft_consensus.cc:3037] T 158f407899b6440cb33d57025ab56873 P f0c82b0e87d84b5eb3b2fd4f3e63ff1b [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:14.875677 24080 catalog_manager.cc:5484] T 19eec94f19284c9e97efdbe449a5238d P 5ce52428c87b46369add4a8afaaef67c reported cstate change: term changed from 1 to 2, leader changed from fb59098726224e7bae383f5c11820b39 (127.21.84.129) to 5ce52428c87b46369add4a8afaaef67c (127.21.84.132). New cstate: current_term: 2 leader_uuid: "5ce52428c87b46369add4a8afaaef67c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ce52428c87b46369add4a8afaaef67c" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 43389 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:14.877125 25451 raft_consensus.cc:504] T 158f407899b6440cb33d57025ab56873 P f0c82b0e87d84b5eb3b2fd4f3e63ff1b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 35031 } attrs { promote: false } }
I20240501 08:11:14.878397 24359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "158f407899b6440cb33d57025ab56873" candidate_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "213ccbb05e6f44deb4a34d0a7041f7de"
I20240501 08:11:14.878635 24359 raft_consensus.cc:3032] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:11:14.878722 24359 raft_consensus.cc:729] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 213ccbb05e6f44deb4a34d0a7041f7de, State: Running, Role: LEADER
I20240501 08:11:14.878998 24359 consensus_queue.cc:260] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 35031 } attrs { promote: false } }
I20240501 08:11:14.879259 24359 raft_consensus.cc:3037] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:14.881121 25451 leader_election.cc:288] T 158f407899b6440cb33d57025ab56873 P f0c82b0e87d84b5eb3b2fd4f3e63ff1b [CANDIDATE]: Term 2 election: Requested vote from peers 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130:37857)
I20240501 08:11:14.883077 24359 raft_consensus.cc:2454] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0c82b0e87d84b5eb3b2fd4f3e63ff1b in term 2.
I20240501 08:11:14.883638 25387 leader_election.cc:302] T 158f407899b6440cb33d57025ab56873 P f0c82b0e87d84b5eb3b2fd4f3e63ff1b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 213ccbb05e6f44deb4a34d0a7041f7de, f0c82b0e87d84b5eb3b2fd4f3e63ff1b; no voters:
I20240501 08:11:14.884135 26255 raft_consensus.cc:2781] T 158f407899b6440cb33d57025ab56873 P f0c82b0e87d84b5eb3b2fd4f3e63ff1b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:11:14.884317 26255 raft_consensus.cc:686] T 158f407899b6440cb33d57025ab56873 P f0c82b0e87d84b5eb3b2fd4f3e63ff1b [term 2 LEADER]: Becoming Leader. State: Replica: f0c82b0e87d84b5eb3b2fd4f3e63ff1b, State: Running, Role: LEADER
I20240501 08:11:14.884577 26255 consensus_queue.cc:237] T 158f407899b6440cb33d57025ab56873 P f0c82b0e87d84b5eb3b2fd4f3e63ff1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 35031 } attrs { promote: false } }
I20240501 08:11:14.887836 24080 catalog_manager.cc:5484] T 158f407899b6440cb33d57025ab56873 P f0c82b0e87d84b5eb3b2fd4f3e63ff1b reported cstate change: term changed from 1 to 2, leader changed from 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130) to f0c82b0e87d84b5eb3b2fd4f3e63ff1b (127.21.84.131). New cstate: current_term: 2 leader_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 35031 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:14.889961 24359 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "158f407899b6440cb33d57025ab56873"
dest_uuid: "213ccbb05e6f44deb4a34d0a7041f7de"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39498
I20240501 08:11:14.890179 24359 raft_consensus.cc:595] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de [term 2 FOLLOWER]: Received request to transfer leadership
I20240501 08:11:14.890276 24359 raft_consensus.cc:603] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240501 08:11:14.901444 24228 raft_consensus.cc:1261] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39 [term 2 FOLLOWER]: Refusing update from remote peer f0c82b0e87d84b5eb3b2fd4f3e63ff1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:11:14.902340 26255 consensus_queue.cc:1035] T 476e8260712e414ab298ea94d413044e P f0c82b0e87d84b5eb3b2fd4f3e63ff1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:11:14.917464 24359 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c4bf2462d8b949d6a2f734780fd0d746"
dest_uuid: "213ccbb05e6f44deb4a34d0a7041f7de"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39518
I20240501 08:11:14.917678 24359 raft_consensus.cc:595] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:14.945255 24359 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0705e5b48e84423fa252da5d62b3008b"
dest_uuid: "213ccbb05e6f44deb4a34d0a7041f7de"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39534
I20240501 08:11:14.945457 24359 raft_consensus.cc:595] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:14.998596 26344 raft_consensus.cc:979] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de: : Instructing follower 8a102eff724b4f459bc91203802847fb to start an election
I20240501 08:11:14.998783 26344 raft_consensus.cc:1067] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Signalling peer 8a102eff724b4f459bc91203802847fb to start an election
I20240501 08:11:14.999418 25972 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c4bf2462d8b949d6a2f734780fd0d746"
dest_uuid: "8a102eff724b4f459bc91203802847fb"
from {username='slave'} at 127.21.84.130:50641
I20240501 08:11:14.999614 25972 raft_consensus.cc:482] T c4bf2462d8b949d6a2f734780fd0d746 P 8a102eff724b4f459bc91203802847fb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:11:14.999720 25972 raft_consensus.cc:3037] T c4bf2462d8b949d6a2f734780fd0d746 P 8a102eff724b4f459bc91203802847fb [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:15.002100 25972 raft_consensus.cc:504] T c4bf2462d8b949d6a2f734780fd0d746 P 8a102eff724b4f459bc91203802847fb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "8a102eff724b4f459bc91203802847fb" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 35569 } attrs { promote: false } }
I20240501 08:11:15.002701 25972 leader_election.cc:288] T c4bf2462d8b949d6a2f734780fd0d746 P 8a102eff724b4f459bc91203802847fb [CANDIDATE]: Term 2 election: Requested vote from peers 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130:37857)
I20240501 08:11:15.003224 24359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4bf2462d8b949d6a2f734780fd0d746" candidate_uuid: "8a102eff724b4f459bc91203802847fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "213ccbb05e6f44deb4a34d0a7041f7de"
I20240501 08:11:15.003399 24359 raft_consensus.cc:3032] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:11:15.003476 24359 raft_consensus.cc:729] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 213ccbb05e6f44deb4a34d0a7041f7de, State: Running, Role: LEADER
I20240501 08:11:15.003700 24359 consensus_queue.cc:260] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "8a102eff724b4f459bc91203802847fb" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 35569 } attrs { promote: false } }
I20240501 08:11:15.003957 24359 raft_consensus.cc:3037] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:15.005841 24359 raft_consensus.cc:2454] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a102eff724b4f459bc91203802847fb in term 2.
I20240501 08:11:15.006314 25908 leader_election.cc:302] T c4bf2462d8b949d6a2f734780fd0d746 P 8a102eff724b4f459bc91203802847fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 213ccbb05e6f44deb4a34d0a7041f7de, 8a102eff724b4f459bc91203802847fb; no voters:
I20240501 08:11:15.006958 26652 raft_consensus.cc:2781] T c4bf2462d8b949d6a2f734780fd0d746 P 8a102eff724b4f459bc91203802847fb [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:11:15.007376 26652 raft_consensus.cc:686] T c4bf2462d8b949d6a2f734780fd0d746 P 8a102eff724b4f459bc91203802847fb [term 2 LEADER]: Becoming Leader. State: Replica: 8a102eff724b4f459bc91203802847fb, State: Running, Role: LEADER
I20240501 08:11:15.007645 26652 consensus_queue.cc:237] T c4bf2462d8b949d6a2f734780fd0d746 P 8a102eff724b4f459bc91203802847fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "8a102eff724b4f459bc91203802847fb" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 35569 } attrs { promote: false } }
I20240501 08:11:15.009797 24080 catalog_manager.cc:5484] T c4bf2462d8b949d6a2f734780fd0d746 P 8a102eff724b4f459bc91203802847fb reported cstate change: term changed from 1 to 2, leader changed from 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130) to 8a102eff724b4f459bc91203802847fb (127.21.84.135). New cstate: current_term: 2 leader_uuid: "8a102eff724b4f459bc91203802847fb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a102eff724b4f459bc91203802847fb" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 35569 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:15.022548 26478 raft_consensus.cc:979] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de: : Instructing follower 1ad8fc587a734cc48f7f49b00fc0c0a1 to start an election
I20240501 08:11:15.022753 26478 raft_consensus.cc:1067] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Signalling peer 1ad8fc587a734cc48f7f49b00fc0c0a1 to start an election
I20240501 08:11:15.023564 25842 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6305a4cfdfc3420594ff185a15723ad1"
dest_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1"
from {username='slave'} at 127.21.84.130:50217
I20240501 08:11:15.023880 25842 raft_consensus.cc:482] T 6305a4cfdfc3420594ff185a15723ad1 P 1ad8fc587a734cc48f7f49b00fc0c0a1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:11:15.024013 25842 raft_consensus.cc:3037] T 6305a4cfdfc3420594ff185a15723ad1 P 1ad8fc587a734cc48f7f49b00fc0c0a1 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:15.027743 25842 raft_consensus.cc:504] T 6305a4cfdfc3420594ff185a15723ad1 P 1ad8fc587a734cc48f7f49b00fc0c0a1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 45565 } attrs { promote: false } }
I20240501 08:11:15.028437 25842 leader_election.cc:288] T 6305a4cfdfc3420594ff185a15723ad1 P 1ad8fc587a734cc48f7f49b00fc0c0a1 [CANDIDATE]: Term 2 election: Requested vote from peers 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130:37857)
I20240501 08:11:15.029124 24359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6305a4cfdfc3420594ff185a15723ad1" candidate_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "213ccbb05e6f44deb4a34d0a7041f7de"
I20240501 08:11:15.029326 24359 raft_consensus.cc:3032] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:11:15.029425 24359 raft_consensus.cc:729] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 213ccbb05e6f44deb4a34d0a7041f7de, State: Running, Role: LEADER
I20240501 08:11:15.029671 24359 consensus_queue.cc:260] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 45565 } attrs { promote: false } }
I20240501 08:11:15.029980 24359 raft_consensus.cc:3037] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:15.031975 24359 raft_consensus.cc:2454] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ad8fc587a734cc48f7f49b00fc0c0a1 in term 2.
I20240501 08:11:15.032450 25778 leader_election.cc:302] T 6305a4cfdfc3420594ff185a15723ad1 P 1ad8fc587a734cc48f7f49b00fc0c0a1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1ad8fc587a734cc48f7f49b00fc0c0a1, 213ccbb05e6f44deb4a34d0a7041f7de; no voters:
I20240501 08:11:15.032805 26323 raft_consensus.cc:2781] T 6305a4cfdfc3420594ff185a15723ad1 P 1ad8fc587a734cc48f7f49b00fc0c0a1 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:11:15.033013 26323 raft_consensus.cc:686] T 6305a4cfdfc3420594ff185a15723ad1 P 1ad8fc587a734cc48f7f49b00fc0c0a1 [term 2 LEADER]: Becoming Leader. State: Replica: 1ad8fc587a734cc48f7f49b00fc0c0a1, State: Running, Role: LEADER
I20240501 08:11:15.033341 26323 consensus_queue.cc:237] T 6305a4cfdfc3420594ff185a15723ad1 P 1ad8fc587a734cc48f7f49b00fc0c0a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 45565 } attrs { promote: false } }
I20240501 08:11:15.036640 24080 catalog_manager.cc:5484] T 6305a4cfdfc3420594ff185a15723ad1 P 1ad8fc587a734cc48f7f49b00fc0c0a1 reported cstate change: term changed from 1 to 2, leader changed from 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130) to 1ad8fc587a734cc48f7f49b00fc0c0a1 (127.21.84.134). New cstate: current_term: 2 leader_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 45565 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:15.038875 26344 raft_consensus.cc:979] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de: : Instructing follower 5ce52428c87b46369add4a8afaaef67c to start an election
I20240501 08:11:15.039033 26344 raft_consensus.cc:1067] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Signalling peer 5ce52428c87b46369add4a8afaaef67c to start an election
I20240501 08:11:15.039775 25582 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0705e5b48e84423fa252da5d62b3008b"
dest_uuid: "5ce52428c87b46369add4a8afaaef67c"
from {username='slave'} at 127.21.84.130:46853
I20240501 08:11:15.040045 25582 raft_consensus.cc:482] T 0705e5b48e84423fa252da5d62b3008b P 5ce52428c87b46369add4a8afaaef67c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:11:15.040180 25582 raft_consensus.cc:3037] T 0705e5b48e84423fa252da5d62b3008b P 5ce52428c87b46369add4a8afaaef67c [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:15.042501 25582 raft_consensus.cc:504] T 0705e5b48e84423fa252da5d62b3008b P 5ce52428c87b46369add4a8afaaef67c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "5ce52428c87b46369add4a8afaaef67c" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 43389 } attrs { promote: false } }
I20240501 08:11:15.043066 25582 leader_election.cc:288] T 0705e5b48e84423fa252da5d62b3008b P 5ce52428c87b46369add4a8afaaef67c [CANDIDATE]: Term 2 election: Requested vote from peers 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130:37857)
I20240501 08:11:15.043684 24359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0705e5b48e84423fa252da5d62b3008b" candidate_uuid: "5ce52428c87b46369add4a8afaaef67c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "213ccbb05e6f44deb4a34d0a7041f7de"
I20240501 08:11:15.043927 24359 raft_consensus.cc:3032] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:11:15.044631 24359 raft_consensus.cc:729] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 213ccbb05e6f44deb4a34d0a7041f7de, State: Running, Role: LEADER
I20240501 08:11:15.044982 24359 consensus_queue.cc:260] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "5ce52428c87b46369add4a8afaaef67c" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 43389 } attrs { promote: false } }
I20240501 08:11:15.045397 24359 raft_consensus.cc:3037] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:15.049152 24359 raft_consensus.cc:2454] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce52428c87b46369add4a8afaaef67c in term 2.
I20240501 08:11:15.049691 25518 leader_election.cc:302] T 0705e5b48e84423fa252da5d62b3008b P 5ce52428c87b46369add4a8afaaef67c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 213ccbb05e6f44deb4a34d0a7041f7de, 5ce52428c87b46369add4a8afaaef67c; no voters:
I20240501 08:11:15.050339 26345 raft_consensus.cc:2781] T 0705e5b48e84423fa252da5d62b3008b P 5ce52428c87b46369add4a8afaaef67c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:11:15.050542 26345 raft_consensus.cc:686] T 0705e5b48e84423fa252da5d62b3008b P 5ce52428c87b46369add4a8afaaef67c [term 2 LEADER]: Becoming Leader. State: Replica: 5ce52428c87b46369add4a8afaaef67c, State: Running, Role: LEADER
I20240501 08:11:15.050850 26345 consensus_queue.cc:237] T 0705e5b48e84423fa252da5d62b3008b P 5ce52428c87b46369add4a8afaaef67c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "5ce52428c87b46369add4a8afaaef67c" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 43389 } attrs { promote: false } }
I20240501 08:11:15.053439 24080 catalog_manager.cc:5484] T 0705e5b48e84423fa252da5d62b3008b P 5ce52428c87b46369add4a8afaaef67c reported cstate change: term changed from 1 to 2, leader changed from 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130) to 5ce52428c87b46369add4a8afaaef67c (127.21.84.132). New cstate: current_term: 2 leader_uuid: "5ce52428c87b46369add4a8afaaef67c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ce52428c87b46369add4a8afaaef67c" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 43389 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:15.087576 24228 raft_consensus.cc:1261] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39 [term 2 FOLLOWER]: Refusing update from remote peer 8a102eff724b4f459bc91203802847fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:11:15.088364 26652 consensus_queue.cc:1035] T abc7faa250d14cecb417da206b1927a3 P 8a102eff724b4f459bc91203802847fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:11:15.094539 25451 consensus_queue.cc:237] T 476e8260712e414ab298ea94d413044e P f0c82b0e87d84b5eb3b2fd4f3e63ff1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 35031 } attrs { promote: false } }
I20240501 08:11:15.096462 26616 raft_consensus.cc:2932] T 476e8260712e414ab298ea94d413044e P f0c82b0e87d84b5eb3b2fd4f3e63ff1b [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER fb59098726224e7bae383f5c11820b39 (127.21.84.129) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 35031 } attrs { promote: false } } }
I20240501 08:11:15.100706 24071 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 476e8260712e414ab298ea94d413044e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 08:11:15.101771 24080 catalog_manager.cc:5484] T 476e8260712e414ab298ea94d413044e P f0c82b0e87d84b5eb3b2fd4f3e63ff1b reported cstate change: config changed from index 5 to 7, VOTER fb59098726224e7bae383f5c11820b39 (127.21.84.129) evicted. New cstate: current_term: 2 leader_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 35031 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:15.105679 24208 tablet_service.cc:1514] Processing DeleteTablet for tablet 476e8260712e414ab298ea94d413044e with delete_type TABLET_DATA_TOMBSTONED (TS fb59098726224e7bae383f5c11820b39 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:46956
I20240501 08:11:15.106498 26656 tablet_replica.cc:330] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39: stopping tablet replica
I20240501 08:11:15.106892 26656 raft_consensus.cc:2229] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:15.107115 26656 raft_consensus.cc:2258] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:15.108573 26656 ts_tablet_manager.cc:1875] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:15.109871 24228 raft_consensus.cc:1261] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39 [term 2 FOLLOWER]: Refusing update from remote peer 962160cac57040678e5cacd4b8b73c4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:11:15.110729 26343 consensus_queue.cc:1035] T b0c941c4d9ba4607b947765c218a1a60 P 962160cac57040678e5cacd4b8b73c4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:11:15.116796 26656 ts_tablet_manager.cc:1888] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 08:11:15.117041 26656 log.cc:1196] T 476e8260712e414ab298ea94d413044e P fb59098726224e7bae383f5c11820b39: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/476e8260712e414ab298ea94d413044e
I20240501 08:11:15.131716 24073 catalog_manager.cc:4830] TS fb59098726224e7bae383f5c11820b39 (127.21.84.129:44759): tablet 476e8260712e414ab298ea94d413044e (table rebalance_test_table_1 [id=60c95048bf784e6ab27a6c2db06a3f90]) successfully deleted
I20240501 08:11:15.136200 25972 consensus_queue.cc:237] T abc7faa250d14cecb417da206b1927a3 P 8a102eff724b4f459bc91203802847fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8a102eff724b4f459bc91203802847fb" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 35569 } attrs { promote: false } }
I20240501 08:11:15.137367 26652 raft_consensus.cc:2932] T abc7faa250d14cecb417da206b1927a3 P 8a102eff724b4f459bc91203802847fb [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER fb59098726224e7bae383f5c11820b39 (127.21.84.129) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8a102eff724b4f459bc91203802847fb" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 35569 } attrs { promote: false } } }
I20240501 08:11:15.141176 24071 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet abc7faa250d14cecb417da206b1927a3 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 08:11:15.143239 24080 catalog_manager.cc:5484] T abc7faa250d14cecb417da206b1927a3 P 8a102eff724b4f459bc91203802847fb reported cstate change: config changed from index 5 to 7, VOTER fb59098726224e7bae383f5c11820b39 (127.21.84.129) evicted. New cstate: current_term: 2 leader_uuid: "8a102eff724b4f459bc91203802847fb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8a102eff724b4f459bc91203802847fb" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 35569 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:15.146701 24208 tablet_service.cc:1514] Processing DeleteTablet for tablet abc7faa250d14cecb417da206b1927a3 with delete_type TABLET_DATA_TOMBSTONED (TS fb59098726224e7bae383f5c11820b39 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:46956
I20240501 08:11:15.147070 26656 tablet_replica.cc:330] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39: stopping tablet replica
I20240501 08:11:15.147245 26656 raft_consensus.cc:2229] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:15.147440 26656 raft_consensus.cc:2258] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:15.148237 26656 ts_tablet_manager.cc:1875] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:15.152654 26656 ts_tablet_manager.cc:1888] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 08:11:15.152768 26656 log.cc:1196] T abc7faa250d14cecb417da206b1927a3 P fb59098726224e7bae383f5c11820b39: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/abc7faa250d14cecb417da206b1927a3
I20240501 08:11:15.153502 24073 catalog_manager.cc:4830] TS fb59098726224e7bae383f5c11820b39 (127.21.84.129:44759): tablet abc7faa250d14cecb417da206b1927a3 (table rebalance_test_table_1 [id=60c95048bf784e6ab27a6c2db06a3f90]) successfully deleted
I20240501 08:11:15.288409 25712 consensus_queue.cc:237] T b0c941c4d9ba4607b947765c218a1a60 P 962160cac57040678e5cacd4b8b73c4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "962160cac57040678e5cacd4b8b73c4b" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 37341 } attrs { promote: false } }
I20240501 08:11:15.289387 26343 raft_consensus.cc:2932] T b0c941c4d9ba4607b947765c218a1a60 P 962160cac57040678e5cacd4b8b73c4b [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER fb59098726224e7bae383f5c11820b39 (127.21.84.129) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "962160cac57040678e5cacd4b8b73c4b" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 37341 } attrs { promote: false } } }
I20240501 08:11:15.297967 24074 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b0c941c4d9ba4607b947765c218a1a60 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 08:11:15.299425 24080 catalog_manager.cc:5484] T b0c941c4d9ba4607b947765c218a1a60 P 962160cac57040678e5cacd4b8b73c4b reported cstate change: config changed from index 3 to 5, VOTER fb59098726224e7bae383f5c11820b39 (127.21.84.129) evicted. New cstate: current_term: 2 leader_uuid: "962160cac57040678e5cacd4b8b73c4b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "962160cac57040678e5cacd4b8b73c4b" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 37341 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:15.303335 24208 tablet_service.cc:1514] Processing DeleteTablet for tablet b0c941c4d9ba4607b947765c218a1a60 with delete_type TABLET_DATA_TOMBSTONED (TS fb59098726224e7bae383f5c11820b39 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46956
I20240501 08:11:15.303704 26656 tablet_replica.cc:330] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39: stopping tablet replica
I20240501 08:11:15.303865 26656 raft_consensus.cc:2229] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:15.304018 26656 raft_consensus.cc:2258] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:15.304931 26656 ts_tablet_manager.cc:1875] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:15.305372 24228 raft_consensus.cc:1261] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39 [term 2 FOLLOWER]: Refusing update from remote peer 5ce52428c87b46369add4a8afaaef67c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:11:15.306095 26345 consensus_queue.cc:1035] T 19eec94f19284c9e97efdbe449a5238d P 5ce52428c87b46369add4a8afaaef67c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:11:15.311244 26656 ts_tablet_manager.cc:1888] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240501 08:11:15.311434 26656 log.cc:1196] T b0c941c4d9ba4607b947765c218a1a60 P fb59098726224e7bae383f5c11820b39: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/b0c941c4d9ba4607b947765c218a1a60
I20240501 08:11:15.312191 24073 catalog_manager.cc:4830] TS fb59098726224e7bae383f5c11820b39 (127.21.84.129:44759): tablet b0c941c4d9ba4607b947765c218a1a60 (table TestTable [id=a447af67cd6347b1bcb9dbae4ec1aedf]) successfully deleted
I20240501 08:11:15.375775 24228 raft_consensus.cc:1261] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39 [term 2 FOLLOWER]: Refusing update from remote peer 962160cac57040678e5cacd4b8b73c4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:11:15.376559 26343 consensus_queue.cc:1035] T 9b065f9074a74fd9945f22596e7594db P 962160cac57040678e5cacd4b8b73c4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:11:15.422426 25712 consensus_queue.cc:237] T 9b065f9074a74fd9945f22596e7594db P 962160cac57040678e5cacd4b8b73c4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "962160cac57040678e5cacd4b8b73c4b" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 37341 } attrs { promote: false } }
I20240501 08:11:15.423336 26343 raft_consensus.cc:2932] T 9b065f9074a74fd9945f22596e7594db P 962160cac57040678e5cacd4b8b73c4b [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER fb59098726224e7bae383f5c11820b39 (127.21.84.129) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "962160cac57040678e5cacd4b8b73c4b" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 37341 } attrs { promote: false } } }
I20240501 08:11:15.426668 24074 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9b065f9074a74fd9945f22596e7594db with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 08:11:15.428320 24078 catalog_manager.cc:5484] T 9b065f9074a74fd9945f22596e7594db P 962160cac57040678e5cacd4b8b73c4b reported cstate change: config changed from index 5 to 7, VOTER fb59098726224e7bae383f5c11820b39 (127.21.84.129) evicted. New cstate: current_term: 2 leader_uuid: "962160cac57040678e5cacd4b8b73c4b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "962160cac57040678e5cacd4b8b73c4b" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 37341 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:15.432695 24208 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b065f9074a74fd9945f22596e7594db with delete_type TABLET_DATA_TOMBSTONED (TS fb59098726224e7bae383f5c11820b39 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:46956
I20240501 08:11:15.433070 26656 tablet_replica.cc:330] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39: stopping tablet replica
I20240501 08:11:15.433271 26656 raft_consensus.cc:2229] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:15.433372 24359 raft_consensus.cc:1261] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de [term 2 FOLLOWER]: Refusing update from remote peer 1ad8fc587a734cc48f7f49b00fc0c0a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:11:15.433468 26656 raft_consensus.cc:2258] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:15.434226 26323 consensus_queue.cc:1035] T 89a8cf2415454cb7b9157703f72913f4 P 1ad8fc587a734cc48f7f49b00fc0c0a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:11:15.434448 26656 ts_tablet_manager.cc:1875] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:15.441298 26656 ts_tablet_manager.cc:1888] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 08:11:15.441488 26656 log.cc:1196] T 9b065f9074a74fd9945f22596e7594db P fb59098726224e7bae383f5c11820b39: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/9b065f9074a74fd9945f22596e7594db
I20240501 08:11:15.442407 24073 catalog_manager.cc:4830] TS fb59098726224e7bae383f5c11820b39 (127.21.84.129:44759): tablet 9b065f9074a74fd9945f22596e7594db (table rebalance_test_table_3 [id=c0139c2c078545309a3a5eef71a69567]) successfully deleted
I20240501 08:11:15.452148 24359 raft_consensus.cc:1261] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de [term 2 FOLLOWER]: Refusing update from remote peer 1ad8fc587a734cc48f7f49b00fc0c0a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:11:15.452797 26323 consensus_queue.cc:1035] T 6305a4cfdfc3420594ff185a15723ad1 P 1ad8fc587a734cc48f7f49b00fc0c0a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:11:15.458853 25842 consensus_queue.cc:237] T 89a8cf2415454cb7b9157703f72913f4 P 1ad8fc587a734cc48f7f49b00fc0c0a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 45565 } attrs { promote: false } }
I20240501 08:11:15.459707 26323 raft_consensus.cc:2932] T 89a8cf2415454cb7b9157703f72913f4 P 1ad8fc587a734cc48f7f49b00fc0c0a1 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 45565 } attrs { promote: false } } }
I20240501 08:11:15.463328 24071 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 89a8cf2415454cb7b9157703f72913f4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 08:11:15.464661 24078 catalog_manager.cc:5484] T 89a8cf2415454cb7b9157703f72913f4 P 1ad8fc587a734cc48f7f49b00fc0c0a1 reported cstate change: config changed from index 5 to 7, VOTER 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130) evicted. New cstate: current_term: 2 leader_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 45565 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:15.468168 24339 tablet_service.cc:1514] Processing DeleteTablet for tablet 89a8cf2415454cb7b9157703f72913f4 with delete_type TABLET_DATA_TOMBSTONED (TS 213ccbb05e6f44deb4a34d0a7041f7de not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52158
I20240501 08:11:15.469126 26779 tablet_replica.cc:330] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de: stopping tablet replica
I20240501 08:11:15.469632 26779 raft_consensus.cc:2229] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:15.469925 26779 raft_consensus.cc:2258] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:15.470911 24359 raft_consensus.cc:1261] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de [term 2 FOLLOWER]: Refusing update from remote peer f0c82b0e87d84b5eb3b2fd4f3e63ff1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:11:15.471500 26616 consensus_queue.cc:1035] T 158f407899b6440cb33d57025ab56873 P f0c82b0e87d84b5eb3b2fd4f3e63ff1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:11:15.471923 26779 ts_tablet_manager.cc:1875] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:15.479951 26779 ts_tablet_manager.cc:1888] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 08:11:15.480273 26779 log.cc:1196] T 89a8cf2415454cb7b9157703f72913f4 P 213ccbb05e6f44deb4a34d0a7041f7de: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/89a8cf2415454cb7b9157703f72913f4
I20240501 08:11:15.481308 24074 catalog_manager.cc:4830] TS 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130:37857): tablet 89a8cf2415454cb7b9157703f72913f4 (table rebalance_test_table_4 [id=d5585afc570243fbbbb154585b168608]) successfully deleted
I20240501 08:11:15.484781 24359 raft_consensus.cc:1261] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de [term 2 FOLLOWER]: Refusing update from remote peer 8a102eff724b4f459bc91203802847fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:11:15.485472 26652 consensus_queue.cc:1035] T c4bf2462d8b949d6a2f734780fd0d746 P 8a102eff724b4f459bc91203802847fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:11:15.486042 24359 raft_consensus.cc:1261] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de [term 2 FOLLOWER]: Refusing update from remote peer 5ce52428c87b46369add4a8afaaef67c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:11:15.486729 26345 consensus_queue.cc:1035] T 0705e5b48e84423fa252da5d62b3008b P 5ce52428c87b46369add4a8afaaef67c [LEADER]: Connected to new peer: Peer: permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:11:15.489852 25451 consensus_queue.cc:237] T 158f407899b6440cb33d57025ab56873 P f0c82b0e87d84b5eb3b2fd4f3e63ff1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 35031 } attrs { promote: false } }
I20240501 08:11:15.491961 26616 raft_consensus.cc:2932] T 158f407899b6440cb33d57025ab56873 P f0c82b0e87d84b5eb3b2fd4f3e63ff1b [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 35031 } attrs { promote: false } } }
I20240501 08:11:15.495676 24071 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 158f407899b6440cb33d57025ab56873 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 08:11:15.498163 24087 catalog_manager.cc:5484] T 158f407899b6440cb33d57025ab56873 P f0c82b0e87d84b5eb3b2fd4f3e63ff1b reported cstate change: config changed from index 5 to 7, VOTER 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130) evicted. New cstate: current_term: 2 leader_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f0c82b0e87d84b5eb3b2fd4f3e63ff1b" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 35031 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:15.502862 24339 tablet_service.cc:1514] Processing DeleteTablet for tablet 158f407899b6440cb33d57025ab56873 with delete_type TABLET_DATA_TOMBSTONED (TS 213ccbb05e6f44deb4a34d0a7041f7de not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52158
I20240501 08:11:15.503211 26779 tablet_replica.cc:330] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de: stopping tablet replica
I20240501 08:11:15.503422 26779 raft_consensus.cc:2229] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:15.503616 26779 raft_consensus.cc:2258] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:15.504734 26779 ts_tablet_manager.cc:1875] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:15.510015 26779 ts_tablet_manager.cc:1888] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 08:11:15.510175 26779 log.cc:1196] T 158f407899b6440cb33d57025ab56873 P 213ccbb05e6f44deb4a34d0a7041f7de: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/158f407899b6440cb33d57025ab56873
I20240501 08:11:15.510926 24074 catalog_manager.cc:4830] TS 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130:37857): tablet 158f407899b6440cb33d57025ab56873 (table rebalance_test_table_4 [id=d5585afc570243fbbbb154585b168608]) successfully deleted
I20240501 08:11:15.523448 25582 consensus_queue.cc:237] T 19eec94f19284c9e97efdbe449a5238d P 5ce52428c87b46369add4a8afaaef67c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5ce52428c87b46369add4a8afaaef67c" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 43389 } attrs { promote: false } }
I20240501 08:11:15.524363 26596 raft_consensus.cc:2932] T 19eec94f19284c9e97efdbe449a5238d P 5ce52428c87b46369add4a8afaaef67c [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER fb59098726224e7bae383f5c11820b39 (127.21.84.129) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5ce52428c87b46369add4a8afaaef67c" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 43389 } attrs { promote: false } } }
I20240501 08:11:15.528321 24072 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 19eec94f19284c9e97efdbe449a5238d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 08:11:15.530200 24087 catalog_manager.cc:5484] T 19eec94f19284c9e97efdbe449a5238d P 5ce52428c87b46369add4a8afaaef67c reported cstate change: config changed from index 5 to 7, VOTER fb59098726224e7bae383f5c11820b39 (127.21.84.129) evicted. New cstate: current_term: 2 leader_uuid: "5ce52428c87b46369add4a8afaaef67c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5ce52428c87b46369add4a8afaaef67c" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 43389 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:15.534338 24208 tablet_service.cc:1514] Processing DeleteTablet for tablet 19eec94f19284c9e97efdbe449a5238d with delete_type TABLET_DATA_TOMBSTONED (TS fb59098726224e7bae383f5c11820b39 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:46956
I20240501 08:11:15.534694 26656 tablet_replica.cc:330] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39: stopping tablet replica
I20240501 08:11:15.534878 26656 raft_consensus.cc:2229] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:15.535063 26656 raft_consensus.cc:2258] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:15.535830 26656 ts_tablet_manager.cc:1875] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:15.541693 26656 ts_tablet_manager.cc:1888] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 08:11:15.541859 26656 log.cc:1196] T 19eec94f19284c9e97efdbe449a5238d P fb59098726224e7bae383f5c11820b39: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/19eec94f19284c9e97efdbe449a5238d
I20240501 08:11:15.542680 24073 catalog_manager.cc:4830] TS fb59098726224e7bae383f5c11820b39 (127.21.84.129:44759): tablet 19eec94f19284c9e97efdbe449a5238d (table rebalance_test_table_0 [id=a0b88ef77ce744fd84b0987b68eda81a]) successfully deleted
I20240501 08:11:15.645123 25971 consensus_queue.cc:237] T c4bf2462d8b949d6a2f734780fd0d746 P 8a102eff724b4f459bc91203802847fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8a102eff724b4f459bc91203802847fb" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 35569 } attrs { promote: false } }
I20240501 08:11:15.646150 26652 raft_consensus.cc:2932] T c4bf2462d8b949d6a2f734780fd0d746 P 8a102eff724b4f459bc91203802847fb [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8a102eff724b4f459bc91203802847fb" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 35569 } attrs { promote: false } } }
I20240501 08:11:15.648764 24071 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c4bf2462d8b949d6a2f734780fd0d746 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 08:11:15.650104 24087 catalog_manager.cc:5484] T c4bf2462d8b949d6a2f734780fd0d746 P 8a102eff724b4f459bc91203802847fb reported cstate change: config changed from index 5 to 7, VOTER 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130) evicted. New cstate: current_term: 2 leader_uuid: "8a102eff724b4f459bc91203802847fb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8a102eff724b4f459bc91203802847fb" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 35569 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:15.653383 24339 tablet_service.cc:1514] Processing DeleteTablet for tablet c4bf2462d8b949d6a2f734780fd0d746 with delete_type TABLET_DATA_TOMBSTONED (TS 213ccbb05e6f44deb4a34d0a7041f7de not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52158
I20240501 08:11:15.653707 26779 tablet_replica.cc:330] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de: stopping tablet replica
I20240501 08:11:15.653903 26779 raft_consensus.cc:2229] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:15.654134 26779 raft_consensus.cc:2258] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:15.654827 26779 ts_tablet_manager.cc:1875] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:15.659436 26779 ts_tablet_manager.cc:1888] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 08:11:15.659552 26779 log.cc:1196] T c4bf2462d8b949d6a2f734780fd0d746 P 213ccbb05e6f44deb4a34d0a7041f7de: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/c4bf2462d8b949d6a2f734780fd0d746
I20240501 08:11:15.660264 24074 catalog_manager.cc:4830] TS 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130:37857): tablet c4bf2462d8b949d6a2f734780fd0d746 (table rebalance_test_table_0 [id=a0b88ef77ce744fd84b0987b68eda81a]) successfully deleted
I20240501 08:11:15.681675 25582 consensus_queue.cc:237] T 0705e5b48e84423fa252da5d62b3008b P 5ce52428c87b46369add4a8afaaef67c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5ce52428c87b46369add4a8afaaef67c" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 43389 } attrs { promote: false } }
I20240501 08:11:15.682601 26596 raft_consensus.cc:2932] T 0705e5b48e84423fa252da5d62b3008b P 5ce52428c87b46369add4a8afaaef67c [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5ce52428c87b46369add4a8afaaef67c" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 43389 } attrs { promote: false } } }
I20240501 08:11:15.685264 24072 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0705e5b48e84423fa252da5d62b3008b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 08:11:15.686622 24087 catalog_manager.cc:5484] T 0705e5b48e84423fa252da5d62b3008b P 5ce52428c87b46369add4a8afaaef67c reported cstate change: config changed from index 5 to 7, VOTER 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130) evicted. New cstate: current_term: 2 leader_uuid: "5ce52428c87b46369add4a8afaaef67c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5ce52428c87b46369add4a8afaaef67c" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 43389 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:15.688629 25842 consensus_queue.cc:237] T 6305a4cfdfc3420594ff185a15723ad1 P 1ad8fc587a734cc48f7f49b00fc0c0a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 45565 } attrs { promote: false } }
I20240501 08:11:15.689471 26323 raft_consensus.cc:2932] T 6305a4cfdfc3420594ff185a15723ad1 P 1ad8fc587a734cc48f7f49b00fc0c0a1 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 45565 } attrs { promote: false } } }
I20240501 08:11:15.690073 24339 tablet_service.cc:1514] Processing DeleteTablet for tablet 0705e5b48e84423fa252da5d62b3008b with delete_type TABLET_DATA_TOMBSTONED (TS 213ccbb05e6f44deb4a34d0a7041f7de not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52158
I20240501 08:11:15.690392 26779 tablet_replica.cc:330] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de: stopping tablet replica
I20240501 08:11:15.690557 26779 raft_consensus.cc:2229] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:15.690728 26779 raft_consensus.cc:2258] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:15.691433 26779 ts_tablet_manager.cc:1875] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:15.692608 24071 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6305a4cfdfc3420594ff185a15723ad1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 08:11:15.693846 24087 catalog_manager.cc:5484] T 6305a4cfdfc3420594ff185a15723ad1 P 1ad8fc587a734cc48f7f49b00fc0c0a1 reported cstate change: config changed from index 5 to 7, VOTER 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130) evicted. New cstate: current_term: 2 leader_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1ad8fc587a734cc48f7f49b00fc0c0a1" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 45565 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:15.697160 24339 tablet_service.cc:1514] Processing DeleteTablet for tablet 6305a4cfdfc3420594ff185a15723ad1 with delete_type TABLET_DATA_TOMBSTONED (TS 213ccbb05e6f44deb4a34d0a7041f7de not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52158
I20240501 08:11:15.697559 26779 ts_tablet_manager.cc:1888] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 08:11:15.697671 26779 log.cc:1196] T 0705e5b48e84423fa252da5d62b3008b P 213ccbb05e6f44deb4a34d0a7041f7de: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/0705e5b48e84423fa252da5d62b3008b
I20240501 08:11:15.698311 26779 tablet_replica.cc:330] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de: stopping tablet replica
I20240501 08:11:15.698354 24074 catalog_manager.cc:4830] TS 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130:37857): tablet 0705e5b48e84423fa252da5d62b3008b (table rebalance_test_table_2 [id=f425dec73cd9491b943f3f9897a84256]) successfully deleted
I20240501 08:11:15.698477 26779 raft_consensus.cc:2229] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:15.698666 26779 raft_consensus.cc:2258] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:15.699404 26779 ts_tablet_manager.cc:1875] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:15.703493 26779 ts_tablet_manager.cc:1888] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 08:11:15.703617 26779 log.cc:1196] T 6305a4cfdfc3420594ff185a15723ad1 P 213ccbb05e6f44deb4a34d0a7041f7de: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/6305a4cfdfc3420594ff185a15723ad1
I20240501 08:11:15.704409 24074 catalog_manager.cc:4830] TS 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130:37857): tablet 6305a4cfdfc3420594ff185a15723ad1 (table rebalance_test_table_2 [id=f425dec73cd9491b943f3f9897a84256]) successfully deleted
I20240501 08:11:15.946153 25822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:15.958519 24339 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:15.960604 25952 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:15.961815 25692 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:15.971771 25431 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:16.002801 25562 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:16.033907 24208 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:16.188738 24339 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:16.190157 25562 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:16.199980 24208 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:16.203418 25692 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:16.205978 25431 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:16.207298 25822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:16.209513 25952 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:16.916577 25822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:16.917155 24339 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:16.931511 25562 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:16.967873 25952 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:16.980239 25692 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:16.991505 24208 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:16.994362 25431 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:17.168989 24339 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:17.185110 25822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:17.185120 25952 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:17.189227 25692 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:17.192179 25562 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:17.223029 24208 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:17.233826 25431 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:17.509780 24339 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:17.510737 25822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:17.518241 25562 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:17.529783 25952 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:17.549474 24208 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:17.550151 25431 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:17.560849 25692 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
1ba60b54c62540a8a64e0fcd6b4eaa8f | 127.21.84.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.21.84.148:38087 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
1ad8fc587a734cc48f7f49b00fc0c0a1 | 127.21.84.134:45565 | HEALTHY | <none> | 2 | 0
213ccbb05e6f44deb4a34d0a7041f7de | 127.21.84.130:37857 | HEALTHY | <none> | 3 | 0
5ce52428c87b46369add4a8afaaef67c | 127.21.84.132:43389 | HEALTHY | <none> | 2 | 0
8a102eff724b4f459bc91203802847fb | 127.21.84.135:35569 | HEALTHY | <none> | 2 | 0
962160cac57040678e5cacd4b8b73c4b | 127.21.84.133:37341 | HEALTHY | <none> | 2 | 0
f0c82b0e87d84b5eb3b2fd4f3e63ff1b | 127.21.84.131:35031 | HEALTHY | <none> | 2 | 0
fb59098726224e7bae383f5c11820b39 | 127.21.84.129:44759 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.21.84.129 | experimental | 127.21.84.129:44759
local_ip_for_outbound_sockets | 127.21.84.130 | experimental | 127.21.84.130:37857
local_ip_for_outbound_sockets | 127.21.84.131 | experimental | 127.21.84.131:35031
local_ip_for_outbound_sockets | 127.21.84.132 | experimental | 127.21.84.132:43389
local_ip_for_outbound_sockets | 127.21.84.133 | experimental | 127.21.84.133:37341
local_ip_for_outbound_sockets | 127.21.84.134 | experimental | 127.21.84.134:45565
local_ip_for_outbound_sockets | 127.21.84.135 | experimental | 127.21.84.135:35569
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.21.84.129:44759
server_dump_info_path | /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.21.84.130:37857
server_dump_info_path | /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.21.84.131:35031
server_dump_info_path | /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.21.84.132:43389
server_dump_info_path | /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.21.84.133:37341
server_dump_info_path | /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.21.84.134:45565
server_dump_info_path | /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.21.84.135:35569
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.21.84.148:38087 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 08:11:17.624689 21842 log_verifier.cc:126] Checking tablet 0705e5b48e84423fa252da5d62b3008b
I20240501 08:11:17.640741 21842 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 0705e5b48e84423fa252da5d62b3008b
I20240501 08:11:17.640903 21842 log_verifier.cc:126] Checking tablet 158f407899b6440cb33d57025ab56873
I20240501 08:11:17.653949 21842 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 158f407899b6440cb33d57025ab56873
I20240501 08:11:17.654069 21842 log_verifier.cc:126] Checking tablet 19eec94f19284c9e97efdbe449a5238d
I20240501 08:11:17.667694 21842 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 19eec94f19284c9e97efdbe449a5238d
I20240501 08:11:17.667811 21842 log_verifier.cc:126] Checking tablet 28f02b777fc8455893ccb4bd4d585952
I20240501 08:11:17.680032 21842 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 28f02b777fc8455893ccb4bd4d585952
I20240501 08:11:17.680140 21842 log_verifier.cc:126] Checking tablet 476e8260712e414ab298ea94d413044e
I20240501 08:11:17.693109 21842 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 476e8260712e414ab298ea94d413044e
I20240501 08:11:17.693251 21842 log_verifier.cc:126] Checking tablet 49cdeb66bf204db3ac2e6c32f0705382
I20240501 08:11:17.705819 21842 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 49cdeb66bf204db3ac2e6c32f0705382
I20240501 08:11:17.705965 21842 log_verifier.cc:126] Checking tablet 6305a4cfdfc3420594ff185a15723ad1
I20240501 08:11:17.718456 21842 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 6305a4cfdfc3420594ff185a15723ad1
I20240501 08:11:17.718578 21842 log_verifier.cc:126] Checking tablet 89a8cf2415454cb7b9157703f72913f4
I20240501 08:11:17.732075 21842 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 89a8cf2415454cb7b9157703f72913f4
I20240501 08:11:17.732189 21842 log_verifier.cc:126] Checking tablet 9b065f9074a74fd9945f22596e7594db
I20240501 08:11:17.746552 21842 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 9b065f9074a74fd9945f22596e7594db
I20240501 08:11:17.746681 21842 log_verifier.cc:126] Checking tablet a331a1e85ff94e5fa93b42583f1668d5
I20240501 08:11:17.759969 21842 log_verifier.cc:177] Verified matching terms for 3 ops in tablet a331a1e85ff94e5fa93b42583f1668d5
I20240501 08:11:17.760097 21842 log_verifier.cc:126] Checking tablet abc7faa250d14cecb417da206b1927a3
I20240501 08:11:17.773658 21842 log_verifier.cc:177] Verified matching terms for 7 ops in tablet abc7faa250d14cecb417da206b1927a3
I20240501 08:11:17.773787 21842 log_verifier.cc:126] Checking tablet b0c941c4d9ba4607b947765c218a1a60
I20240501 08:11:17.783490 21842 log_verifier.cc:177] Verified matching terms for 5 ops in tablet b0c941c4d9ba4607b947765c218a1a60
I20240501 08:11:17.783638 21842 log_verifier.cc:126] Checking tablet c3651f31063f43dd90576200ec3acbe5
I20240501 08:11:17.797040 21842 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c3651f31063f43dd90576200ec3acbe5
I20240501 08:11:17.797166 21842 log_verifier.cc:126] Checking tablet c4bf2462d8b949d6a2f734780fd0d746
I20240501 08:11:17.810379 21842 log_verifier.cc:177] Verified matching terms for 7 ops in tablet c4bf2462d8b949d6a2f734780fd0d746
I20240501 08:11:17.810525 21842 log_verifier.cc:126] Checking tablet c5b68189903e46e48fde14764c024bff
I20240501 08:11:17.824877 21842 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c5b68189903e46e48fde14764c024bff
I20240501 08:11:17.824999 21842 log_verifier.cc:126] Checking tablet dc01c06c62234c8e96be1da7795149c4
I20240501 08:11:17.838706 21842 log_verifier.cc:177] Verified matching terms for 3 ops in tablet dc01c06c62234c8e96be1da7795149c4
I20240501 08:11:17.841780 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.136:0
--local_ip_for_outbound_sockets=127.21.84.136
--webserver_interface=127.21.84.136
--webserver_port=0
--tserver_master_addrs=127.21.84.190:43065
--builtin_ntp_servers=127.21.84.148:38087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 08:11:18.203068 27030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:18.203377 27030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:18.203487 27030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:18.211627 27030 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:18.211761 27030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:18.211921 27030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.136
I20240501 08:11:18.221676 27030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:38087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.21.84.136
--webserver_port=0
--tserver_master_addrs=127.21.84.190:43065
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.136
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:18.223161 27030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:18.224921 27030 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:18.238135 27035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:18.238876 27036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:18.239470 27039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:18.240372 27030 server_base.cc:1034] running on GCE node
I20240501 08:11:18.241017 27030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:18.242054 27030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:18.243264 27030 hybrid_clock.cc:648] HybridClock initialized: now 1714551078243204 us; error 52 us; skew 500 ppm
I20240501 08:11:18.243624 27030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:18.246044 27030 webserver.cc:460] Webserver started at http://127.21.84.136:35453/ using document root <none> and password file <none>
I20240501 08:11:18.246832 27030 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:18.246945 27030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:18.247265 27030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:18.250218 27030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "d182e91f1aca4b21b897714bd8522dcc"
format_stamp: "Formatted at 2024-05-01 08:11:18 on dist-test-slave-9cgk"
I20240501 08:11:18.251020 27030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "d182e91f1aca4b21b897714bd8522dcc"
format_stamp: "Formatted at 2024-05-01 08:11:18 on dist-test-slave-9cgk"
I20240501 08:11:18.256515 27030 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 08:11:18.260777 27045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:18.262269 27030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:11:18.262435 27030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "d182e91f1aca4b21b897714bd8522dcc"
format_stamp: "Formatted at 2024-05-01 08:11:18 on dist-test-slave-9cgk"
I20240501 08:11:18.262653 27030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:18.290000 27030 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:18.290932 27030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:18.291237 27030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:18.292341 27030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:18.294191 27030 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:18.294292 27030 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:18.294404 27030 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:18.294473 27030 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:18.340567 27030 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.136:40403
I20240501 08:11:18.340624 27158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.136:40403 every 8 connection(s)
I20240501 08:11:18.341893 27030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 08:11:18.343042 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 27030
I20240501 08:11:18.343279 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 08:11:18.359163 27159 heartbeater.cc:347] Connected to a master server at 127.21.84.190:43065
I20240501 08:11:18.359524 27159 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:18.360512 27159 heartbeater.cc:511] Master 127.21.84.190:43065 requested a full tablet report, sending...
I20240501 08:11:18.362421 24081 ts_manager.cc:194] Registered new tserver with Master: d182e91f1aca4b21b897714bd8522dcc (127.21.84.136:40403)
I20240501 08:11:18.363721 24081 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.136:46401
I20240501 08:11:18.867655 25822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:18.877179 25562 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:18.880317 24339 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:18.900470 25692 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:18.916649 27093 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:18.949766 25952 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:18.974138 25431 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:19.014309 24208 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:19.181231 25822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:19.191350 24339 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:19.194636 25562 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:19.195060 25692 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:19.196489 25952 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:19.252377 27093 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:19.268057 25431 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:19.280386 24208 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:19.343683 24228 consensus_queue.cc:237] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "d182e91f1aca4b21b897714bd8522dcc" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 40403 } attrs { promote: true } }
I20240501 08:11:19.346226 27251 raft_consensus.cc:2932] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d182e91f1aca4b21b897714bd8522dcc (127.21.84.136) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "d182e91f1aca4b21b897714bd8522dcc" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 40403 } attrs { promote: true } } }
I20240501 08:11:19.352748 24082 catalog_manager.cc:5484] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39 reported cstate change: config changed from index -1 to 4, NON_VOTER d182e91f1aca4b21b897714bd8522dcc (127.21.84.136) added. New cstate: current_term: 1 leader_uuid: "fb59098726224e7bae383f5c11820b39" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d182e91f1aca4b21b897714bd8522dcc" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 40403 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:19.366897 27159 heartbeater.cc:503] Master 127.21.84.190:43065 was elected leader, sending a full tablet report...
W20240501 08:11:19.374639 24161 consensus_peers.cc:485] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39 -> Peer d182e91f1aca4b21b897714bd8522dcc (127.21.84.136:40403): Couldn't send request to peer d182e91f1aca4b21b897714bd8522dcc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49cdeb66bf204db3ac2e6c32f0705382. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:19.383579 24359 consensus_queue.cc:237] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "d182e91f1aca4b21b897714bd8522dcc" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 40403 } attrs { promote: true } }
I20240501 08:11:19.385391 27265 raft_consensus.cc:2932] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d182e91f1aca4b21b897714bd8522dcc (127.21.84.136) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "d182e91f1aca4b21b897714bd8522dcc" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 40403 } attrs { promote: true } } }
I20240501 08:11:19.391635 24087 catalog_manager.cc:5484] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de reported cstate change: config changed from index -1 to 4, NON_VOTER d182e91f1aca4b21b897714bd8522dcc (127.21.84.136) added. New cstate: current_term: 1 leader_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d182e91f1aca4b21b897714bd8522dcc" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 40403 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 08:11:19.395228 24292 consensus_peers.cc:485] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de -> Peer d182e91f1aca4b21b897714bd8522dcc (127.21.84.136:40403): Couldn't send request to peer d182e91f1aca4b21b897714bd8522dcc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a331a1e85ff94e5fa93b42583f1668d5. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:19.868566 27299 ts_tablet_manager.cc:902] T 49cdeb66bf204db3ac2e6c32f0705382 P d182e91f1aca4b21b897714bd8522dcc: Initiating tablet copy from peer fb59098726224e7bae383f5c11820b39 (127.21.84.129:44759)
I20240501 08:11:19.870373 27299 tablet_copy_client.cc:308] T 49cdeb66bf204db3ac2e6c32f0705382 P d182e91f1aca4b21b897714bd8522dcc: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.129:44759
I20240501 08:11:19.877027 24248 tablet_copy_service.cc:140] P fb59098726224e7bae383f5c11820b39: Received BeginTabletCopySession request for tablet 49cdeb66bf204db3ac2e6c32f0705382 from peer d182e91f1aca4b21b897714bd8522dcc ({username='slave'} at 127.21.84.136:58715)
I20240501 08:11:19.877182 24248 tablet_copy_service.cc:161] P fb59098726224e7bae383f5c11820b39: Beginning new tablet copy session on tablet 49cdeb66bf204db3ac2e6c32f0705382 from peer d182e91f1aca4b21b897714bd8522dcc at {username='slave'} at 127.21.84.136:58715: session id = d182e91f1aca4b21b897714bd8522dcc-49cdeb66bf204db3ac2e6c32f0705382
I20240501 08:11:19.879230 24248 tablet_copy_source_session.cc:205] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:19.883713 27299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49cdeb66bf204db3ac2e6c32f0705382. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:19.891957 27305 ts_tablet_manager.cc:902] T a331a1e85ff94e5fa93b42583f1668d5 P d182e91f1aca4b21b897714bd8522dcc: Initiating tablet copy from peer 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130:37857)
I20240501 08:11:19.893805 27305 tablet_copy_client.cc:308] T a331a1e85ff94e5fa93b42583f1668d5 P d182e91f1aca4b21b897714bd8522dcc: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.130:37857
I20240501 08:11:19.895751 27299 tablet_copy_client.cc:783] T 49cdeb66bf204db3ac2e6c32f0705382 P d182e91f1aca4b21b897714bd8522dcc: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:19.896127 27299 tablet_copy_client.cc:647] T 49cdeb66bf204db3ac2e6c32f0705382 P d182e91f1aca4b21b897714bd8522dcc: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:19.898382 27299 tablet_copy_client.cc:523] T 49cdeb66bf204db3ac2e6c32f0705382 P d182e91f1aca4b21b897714bd8522dcc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:19.900053 24379 tablet_copy_service.cc:140] P 213ccbb05e6f44deb4a34d0a7041f7de: Received BeginTabletCopySession request for tablet a331a1e85ff94e5fa93b42583f1668d5 from peer d182e91f1aca4b21b897714bd8522dcc ({username='slave'} at 127.21.84.136:47079)
I20240501 08:11:19.900220 24379 tablet_copy_service.cc:161] P 213ccbb05e6f44deb4a34d0a7041f7de: Beginning new tablet copy session on tablet a331a1e85ff94e5fa93b42583f1668d5 from peer d182e91f1aca4b21b897714bd8522dcc at {username='slave'} at 127.21.84.136:47079: session id = d182e91f1aca4b21b897714bd8522dcc-a331a1e85ff94e5fa93b42583f1668d5
I20240501 08:11:19.901996 27299 tablet_bootstrap.cc:492] T 49cdeb66bf204db3ac2e6c32f0705382 P d182e91f1aca4b21b897714bd8522dcc: Bootstrap starting.
I20240501 08:11:19.902155 24379 tablet_copy_source_session.cc:205] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:19.903313 27305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a331a1e85ff94e5fa93b42583f1668d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:19.908200 27305 tablet_copy_client.cc:783] T a331a1e85ff94e5fa93b42583f1668d5 P d182e91f1aca4b21b897714bd8522dcc: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:19.908489 27305 tablet_copy_client.cc:647] T a331a1e85ff94e5fa93b42583f1668d5 P d182e91f1aca4b21b897714bd8522dcc: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:19.910343 27305 tablet_copy_client.cc:523] T a331a1e85ff94e5fa93b42583f1668d5 P d182e91f1aca4b21b897714bd8522dcc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:19.912480 27299 log.cc:824] T 49cdeb66bf204db3ac2e6c32f0705382 P d182e91f1aca4b21b897714bd8522dcc: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:19.912722 27305 tablet_bootstrap.cc:492] T a331a1e85ff94e5fa93b42583f1668d5 P d182e91f1aca4b21b897714bd8522dcc: Bootstrap starting.
I20240501 08:11:19.946837 27299 tablet_bootstrap.cc:492] T 49cdeb66bf204db3ac2e6c32f0705382 P d182e91f1aca4b21b897714bd8522dcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:19.946975 27305 tablet_bootstrap.cc:492] T a331a1e85ff94e5fa93b42583f1668d5 P d182e91f1aca4b21b897714bd8522dcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:19.947443 27305 tablet_bootstrap.cc:492] T a331a1e85ff94e5fa93b42583f1668d5 P d182e91f1aca4b21b897714bd8522dcc: Bootstrap complete.
I20240501 08:11:19.947496 27299 tablet_bootstrap.cc:492] T 49cdeb66bf204db3ac2e6c32f0705382 P d182e91f1aca4b21b897714bd8522dcc: Bootstrap complete.
I20240501 08:11:19.947909 27299 ts_tablet_manager.cc:1367] T 49cdeb66bf204db3ac2e6c32f0705382 P d182e91f1aca4b21b897714bd8522dcc: Time spent bootstrapping tablet: real 0.046s user 0.044s sys 0.005s
I20240501 08:11:19.947916 27305 ts_tablet_manager.cc:1367] T a331a1e85ff94e5fa93b42583f1668d5 P d182e91f1aca4b21b897714bd8522dcc: Time spent bootstrapping tablet: real 0.035s user 0.026s sys 0.009s
I20240501 08:11:19.953153 27305 raft_consensus.cc:348] T a331a1e85ff94e5fa93b42583f1668d5 P d182e91f1aca4b21b897714bd8522dcc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "d182e91f1aca4b21b897714bd8522dcc" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 40403 } attrs { promote: true } }
I20240501 08:11:19.953155 27299 raft_consensus.cc:348] T 49cdeb66bf204db3ac2e6c32f0705382 P d182e91f1aca4b21b897714bd8522dcc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "d182e91f1aca4b21b897714bd8522dcc" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 40403 } attrs { promote: true } }
I20240501 08:11:19.953611 27299 raft_consensus.cc:729] T 49cdeb66bf204db3ac2e6c32f0705382 P d182e91f1aca4b21b897714bd8522dcc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d182e91f1aca4b21b897714bd8522dcc, State: Initialized, Role: LEARNER
I20240501 08:11:19.953619 27305 raft_consensus.cc:729] T a331a1e85ff94e5fa93b42583f1668d5 P d182e91f1aca4b21b897714bd8522dcc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d182e91f1aca4b21b897714bd8522dcc, State: Initialized, Role: LEARNER
I20240501 08:11:19.954221 27299 consensus_queue.cc:260] T 49cdeb66bf204db3ac2e6c32f0705382 P d182e91f1aca4b21b897714bd8522dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "d182e91f1aca4b21b897714bd8522dcc" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 40403 } attrs { promote: true } }
I20240501 08:11:19.954213 27305 consensus_queue.cc:260] T a331a1e85ff94e5fa93b42583f1668d5 P d182e91f1aca4b21b897714bd8522dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "d182e91f1aca4b21b897714bd8522dcc" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 40403 } attrs { promote: true } }
I20240501 08:11:19.955909 27299 ts_tablet_manager.cc:1398] T 49cdeb66bf204db3ac2e6c32f0705382 P d182e91f1aca4b21b897714bd8522dcc: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240501 08:11:19.955940 27305 ts_tablet_manager.cc:1398] T a331a1e85ff94e5fa93b42583f1668d5 P d182e91f1aca4b21b897714bd8522dcc: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240501 08:11:19.957000 24248 tablet_copy_service.cc:342] P fb59098726224e7bae383f5c11820b39: Request end of tablet copy session d182e91f1aca4b21b897714bd8522dcc-49cdeb66bf204db3ac2e6c32f0705382 received from {username='slave'} at 127.21.84.136:58715
I20240501 08:11:19.957165 24248 tablet_copy_service.cc:433] P fb59098726224e7bae383f5c11820b39: ending tablet copy session d182e91f1aca4b21b897714bd8522dcc-49cdeb66bf204db3ac2e6c32f0705382 on tablet 49cdeb66bf204db3ac2e6c32f0705382 with peer d182e91f1aca4b21b897714bd8522dcc
I20240501 08:11:19.957185 24379 tablet_copy_service.cc:342] P 213ccbb05e6f44deb4a34d0a7041f7de: Request end of tablet copy session d182e91f1aca4b21b897714bd8522dcc-a331a1e85ff94e5fa93b42583f1668d5 received from {username='slave'} at 127.21.84.136:47079
I20240501 08:11:19.957304 24379 tablet_copy_service.cc:433] P 213ccbb05e6f44deb4a34d0a7041f7de: ending tablet copy session d182e91f1aca4b21b897714bd8522dcc-a331a1e85ff94e5fa93b42583f1668d5 on tablet a331a1e85ff94e5fa93b42583f1668d5 with peer d182e91f1aca4b21b897714bd8522dcc
I20240501 08:11:20.259984 27113 raft_consensus.cc:1203] T 49cdeb66bf204db3ac2e6c32f0705382 P d182e91f1aca4b21b897714bd8522dcc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:11:20.323405 27113 raft_consensus.cc:1203] T a331a1e85ff94e5fa93b42583f1668d5 P d182e91f1aca4b21b897714bd8522dcc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:11:20.768126 27358 raft_consensus.cc:1050] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39: attempting to promote NON_VOTER d182e91f1aca4b21b897714bd8522dcc to VOTER
I20240501 08:11:20.768707 27358 consensus_queue.cc:237] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "d182e91f1aca4b21b897714bd8522dcc" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 40403 } attrs { promote: false } }
I20240501 08:11:20.770562 27113 raft_consensus.cc:1261] T 49cdeb66bf204db3ac2e6c32f0705382 P d182e91f1aca4b21b897714bd8522dcc [term 1 LEARNER]: Refusing update from remote peer fb59098726224e7bae383f5c11820b39: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:11:20.771142 27358 consensus_queue.cc:1035] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d182e91f1aca4b21b897714bd8522dcc" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 40403 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:11:20.774610 27359 raft_consensus.cc:2932] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d182e91f1aca4b21b897714bd8522dcc (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "d182e91f1aca4b21b897714bd8522dcc" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 40403 } attrs { promote: false } } }
I20240501 08:11:20.775357 27113 raft_consensus.cc:2932] T 49cdeb66bf204db3ac2e6c32f0705382 P d182e91f1aca4b21b897714bd8522dcc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d182e91f1aca4b21b897714bd8522dcc (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "d182e91f1aca4b21b897714bd8522dcc" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 40403 } attrs { promote: false } } }
I20240501 08:11:20.779434 24087 catalog_manager.cc:5484] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39 reported cstate change: config changed from index 4 to 5, d182e91f1aca4b21b897714bd8522dcc (127.21.84.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fb59098726224e7bae383f5c11820b39" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d182e91f1aca4b21b897714bd8522dcc" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 40403 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:20.806809 27337 raft_consensus.cc:1050] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de: attempting to promote NON_VOTER d182e91f1aca4b21b897714bd8522dcc to VOTER
I20240501 08:11:20.807368 27337 consensus_queue.cc:237] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "d182e91f1aca4b21b897714bd8522dcc" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 40403 } attrs { promote: false } }
I20240501 08:11:20.809201 27113 raft_consensus.cc:1261] T a331a1e85ff94e5fa93b42583f1668d5 P d182e91f1aca4b21b897714bd8522dcc [term 1 LEARNER]: Refusing update from remote peer 213ccbb05e6f44deb4a34d0a7041f7de: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:11:20.809679 27336 consensus_queue.cc:1035] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de [LEADER]: Connected to new peer: Peer: permanent_uuid: "d182e91f1aca4b21b897714bd8522dcc" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 40403 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:11:20.812005 27337 raft_consensus.cc:2932] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d182e91f1aca4b21b897714bd8522dcc (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "d182e91f1aca4b21b897714bd8522dcc" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 40403 } attrs { promote: false } } }
I20240501 08:11:20.812547 27113 raft_consensus.cc:2932] T a331a1e85ff94e5fa93b42583f1668d5 P d182e91f1aca4b21b897714bd8522dcc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d182e91f1aca4b21b897714bd8522dcc (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "d182e91f1aca4b21b897714bd8522dcc" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 40403 } attrs { promote: false } } }
I20240501 08:11:20.819087 24082 catalog_manager.cc:5484] T a331a1e85ff94e5fa93b42583f1668d5 P d182e91f1aca4b21b897714bd8522dcc reported cstate change: config changed from index 4 to 5, d182e91f1aca4b21b897714bd8522dcc (127.21.84.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "d182e91f1aca4b21b897714bd8522dcc" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 40403 } attrs { promote: false } } }
I20240501 08:11:20.949432 24359 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a331a1e85ff94e5fa93b42583f1668d5"
dest_uuid: "213ccbb05e6f44deb4a34d0a7041f7de"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36468
I20240501 08:11:20.949688 24359 raft_consensus.cc:595] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:20.968339 24228 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "49cdeb66bf204db3ac2e6c32f0705382"
dest_uuid: "fb59098726224e7bae383f5c11820b39"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55412
I20240501 08:11:20.968562 24228 raft_consensus.cc:595] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:21.187203 24359 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a331a1e85ff94e5fa93b42583f1668d5"
dest_uuid: "213ccbb05e6f44deb4a34d0a7041f7de"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36496
I20240501 08:11:21.187423 24359 raft_consensus.cc:595] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:21.204612 24228 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "49cdeb66bf204db3ac2e6c32f0705382"
dest_uuid: "fb59098726224e7bae383f5c11820b39"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55430
I20240501 08:11:21.204854 24228 raft_consensus.cc:595] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:21.232451 27358 raft_consensus.cc:979] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39: : Instructing follower d182e91f1aca4b21b897714bd8522dcc to start an election
I20240501 08:11:21.232659 27358 raft_consensus.cc:1067] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Signalling peer d182e91f1aca4b21b897714bd8522dcc to start an election
I20240501 08:11:21.233348 27113 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "49cdeb66bf204db3ac2e6c32f0705382"
dest_uuid: "d182e91f1aca4b21b897714bd8522dcc"
from {username='slave'} at 127.21.84.129:40141
I20240501 08:11:21.233639 27113 raft_consensus.cc:482] T 49cdeb66bf204db3ac2e6c32f0705382 P d182e91f1aca4b21b897714bd8522dcc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:11:21.233819 27113 raft_consensus.cc:3037] T 49cdeb66bf204db3ac2e6c32f0705382 P d182e91f1aca4b21b897714bd8522dcc [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:21.235801 27113 raft_consensus.cc:504] T 49cdeb66bf204db3ac2e6c32f0705382 P d182e91f1aca4b21b897714bd8522dcc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "d182e91f1aca4b21b897714bd8522dcc" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 40403 } attrs { promote: false } }
I20240501 08:11:21.237566 27113 leader_election.cc:288] T 49cdeb66bf204db3ac2e6c32f0705382 P d182e91f1aca4b21b897714bd8522dcc [CANDIDATE]: Term 2 election: Requested vote from peers fb59098726224e7bae383f5c11820b39 (127.21.84.129:44759)
I20240501 08:11:21.238123 24228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cdeb66bf204db3ac2e6c32f0705382" candidate_uuid: "d182e91f1aca4b21b897714bd8522dcc" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "fb59098726224e7bae383f5c11820b39"
I20240501 08:11:21.238339 24228 raft_consensus.cc:3032] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:11:21.238404 24228 raft_consensus.cc:729] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb59098726224e7bae383f5c11820b39, State: Running, Role: LEADER
I20240501 08:11:21.238657 24228 consensus_queue.cc:260] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "d182e91f1aca4b21b897714bd8522dcc" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 40403 } attrs { promote: false } }
I20240501 08:11:21.238950 24228 raft_consensus.cc:3037] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:21.240823 24228 raft_consensus.cc:2454] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d182e91f1aca4b21b897714bd8522dcc in term 2.
I20240501 08:11:21.241372 27048 leader_election.cc:302] T 49cdeb66bf204db3ac2e6c32f0705382 P d182e91f1aca4b21b897714bd8522dcc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: d182e91f1aca4b21b897714bd8522dcc, fb59098726224e7bae383f5c11820b39; no voters:
I20240501 08:11:21.241721 27313 raft_consensus.cc:2781] T 49cdeb66bf204db3ac2e6c32f0705382 P d182e91f1aca4b21b897714bd8522dcc [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:11:21.242185 27313 raft_consensus.cc:686] T 49cdeb66bf204db3ac2e6c32f0705382 P d182e91f1aca4b21b897714bd8522dcc [term 2 LEADER]: Becoming Leader. State: Replica: d182e91f1aca4b21b897714bd8522dcc, State: Running, Role: LEADER
I20240501 08:11:21.242668 27313 consensus_queue.cc:237] T 49cdeb66bf204db3ac2e6c32f0705382 P d182e91f1aca4b21b897714bd8522dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } } peers { permanent_uuid: "d182e91f1aca4b21b897714bd8522dcc" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 40403 } attrs { promote: false } }
I20240501 08:11:21.246022 24087 catalog_manager.cc:5484] T 49cdeb66bf204db3ac2e6c32f0705382 P d182e91f1aca4b21b897714bd8522dcc reported cstate change: term changed from 1 to 2, leader changed from fb59098726224e7bae383f5c11820b39 (127.21.84.129) to d182e91f1aca4b21b897714bd8522dcc (127.21.84.136). New cstate: current_term: 2 leader_uuid: "d182e91f1aca4b21b897714bd8522dcc" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d182e91f1aca4b21b897714bd8522dcc" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 40403 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:21.251585 27265 raft_consensus.cc:979] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de: : Instructing follower d182e91f1aca4b21b897714bd8522dcc to start an election
I20240501 08:11:21.251767 27265 raft_consensus.cc:1067] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Signalling peer d182e91f1aca4b21b897714bd8522dcc to start an election
I20240501 08:11:21.252411 27113 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a331a1e85ff94e5fa93b42583f1668d5"
dest_uuid: "d182e91f1aca4b21b897714bd8522dcc"
from {username='slave'} at 127.21.84.130:51011
I20240501 08:11:21.252655 27113 raft_consensus.cc:482] T a331a1e85ff94e5fa93b42583f1668d5 P d182e91f1aca4b21b897714bd8522dcc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:11:21.252758 27113 raft_consensus.cc:3037] T a331a1e85ff94e5fa93b42583f1668d5 P d182e91f1aca4b21b897714bd8522dcc [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:21.255482 27113 raft_consensus.cc:504] T a331a1e85ff94e5fa93b42583f1668d5 P d182e91f1aca4b21b897714bd8522dcc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "d182e91f1aca4b21b897714bd8522dcc" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 40403 } attrs { promote: false } }
I20240501 08:11:21.256150 27113 leader_election.cc:288] T a331a1e85ff94e5fa93b42583f1668d5 P d182e91f1aca4b21b897714bd8522dcc [CANDIDATE]: Term 2 election: Requested vote from peers 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130:37857)
W20240501 08:11:21.256620 24071 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 49cdeb66bf204db3ac2e6c32f0705382 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader d182e91f1aca4b21b897714bd8522dcc (127.21.84.136:40403) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240501 08:11:21.256654 24359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a331a1e85ff94e5fa93b42583f1668d5" candidate_uuid: "d182e91f1aca4b21b897714bd8522dcc" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "213ccbb05e6f44deb4a34d0a7041f7de"
I20240501 08:11:21.256858 24359 raft_consensus.cc:3032] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:11:21.256980 24359 raft_consensus.cc:729] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 213ccbb05e6f44deb4a34d0a7041f7de, State: Running, Role: LEADER
I20240501 08:11:21.257689 24359 consensus_queue.cc:260] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "d182e91f1aca4b21b897714bd8522dcc" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 40403 } attrs { promote: false } }
I20240501 08:11:21.258030 24359 raft_consensus.cc:3037] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:21.259804 24359 raft_consensus.cc:2454] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d182e91f1aca4b21b897714bd8522dcc in term 2.
I20240501 08:11:21.260259 27049 leader_election.cc:302] T a331a1e85ff94e5fa93b42583f1668d5 P d182e91f1aca4b21b897714bd8522dcc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 213ccbb05e6f44deb4a34d0a7041f7de, d182e91f1aca4b21b897714bd8522dcc; no voters:
I20240501 08:11:21.260548 27313 raft_consensus.cc:2781] T a331a1e85ff94e5fa93b42583f1668d5 P d182e91f1aca4b21b897714bd8522dcc [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:11:21.260697 27313 raft_consensus.cc:686] T a331a1e85ff94e5fa93b42583f1668d5 P d182e91f1aca4b21b897714bd8522dcc [term 2 LEADER]: Becoming Leader. State: Replica: d182e91f1aca4b21b897714bd8522dcc, State: Running, Role: LEADER
I20240501 08:11:21.260928 27313 consensus_queue.cc:237] T a331a1e85ff94e5fa93b42583f1668d5 P d182e91f1aca4b21b897714bd8522dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } } peers { permanent_uuid: "d182e91f1aca4b21b897714bd8522dcc" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 40403 } attrs { promote: false } }
I20240501 08:11:21.262967 24087 catalog_manager.cc:5484] T a331a1e85ff94e5fa93b42583f1668d5 P d182e91f1aca4b21b897714bd8522dcc reported cstate change: term changed from 1 to 2, leader changed from 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130) to d182e91f1aca4b21b897714bd8522dcc (127.21.84.136). New cstate: current_term: 2 leader_uuid: "d182e91f1aca4b21b897714bd8522dcc" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d182e91f1aca4b21b897714bd8522dcc" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 40403 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:21.675218 24359 raft_consensus.cc:1261] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de [term 2 FOLLOWER]: Refusing update from remote peer d182e91f1aca4b21b897714bd8522dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:11:21.675923 27313 consensus_queue.cc:1035] T a331a1e85ff94e5fa93b42583f1668d5 P d182e91f1aca4b21b897714bd8522dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "213ccbb05e6f44deb4a34d0a7041f7de" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 37857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:11:21.677301 24228 raft_consensus.cc:1261] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39 [term 2 FOLLOWER]: Refusing update from remote peer d182e91f1aca4b21b897714bd8522dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:11:21.678025 27410 consensus_queue.cc:1035] T 49cdeb66bf204db3ac2e6c32f0705382 P d182e91f1aca4b21b897714bd8522dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb59098726224e7bae383f5c11820b39" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 44759 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:11:21.882211 27112 consensus_queue.cc:237] T a331a1e85ff94e5fa93b42583f1668d5 P d182e91f1aca4b21b897714bd8522dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d182e91f1aca4b21b897714bd8522dcc" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 40403 } attrs { promote: false } }
I20240501 08:11:21.883106 27434 raft_consensus.cc:2932] T a331a1e85ff94e5fa93b42583f1668d5 P d182e91f1aca4b21b897714bd8522dcc [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d182e91f1aca4b21b897714bd8522dcc" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 40403 } attrs { promote: false } } }
I20240501 08:11:21.886004 24071 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a331a1e85ff94e5fa93b42583f1668d5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 08:11:21.887450 24087 catalog_manager.cc:5484] T a331a1e85ff94e5fa93b42583f1668d5 P d182e91f1aca4b21b897714bd8522dcc reported cstate change: config changed from index 5 to 7, VOTER 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130) evicted. New cstate: current_term: 2 leader_uuid: "d182e91f1aca4b21b897714bd8522dcc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d182e91f1aca4b21b897714bd8522dcc" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 40403 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:21.891849 24339 tablet_service.cc:1514] Processing DeleteTablet for tablet a331a1e85ff94e5fa93b42583f1668d5 with delete_type TABLET_DATA_TOMBSTONED (TS 213ccbb05e6f44deb4a34d0a7041f7de not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52158
I20240501 08:11:21.892346 27112 consensus_queue.cc:237] T 49cdeb66bf204db3ac2e6c32f0705382 P d182e91f1aca4b21b897714bd8522dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d182e91f1aca4b21b897714bd8522dcc" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 40403 } attrs { promote: false } }
I20240501 08:11:21.892436 27451 tablet_replica.cc:330] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de: stopping tablet replica
I20240501 08:11:21.892612 27451 raft_consensus.cc:2229] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:21.892802 27451 raft_consensus.cc:2258] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:21.893216 27434 raft_consensus.cc:2932] T 49cdeb66bf204db3ac2e6c32f0705382 P d182e91f1aca4b21b897714bd8522dcc [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER fb59098726224e7bae383f5c11820b39 (127.21.84.129) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d182e91f1aca4b21b897714bd8522dcc" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 40403 } attrs { promote: false } } }
I20240501 08:11:21.894658 27451 ts_tablet_manager.cc:1875] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:21.896960 24071 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 49cdeb66bf204db3ac2e6c32f0705382 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 08:11:21.898381 24087 catalog_manager.cc:5484] T 49cdeb66bf204db3ac2e6c32f0705382 P d182e91f1aca4b21b897714bd8522dcc reported cstate change: config changed from index 5 to 7, VOTER fb59098726224e7bae383f5c11820b39 (127.21.84.129) evicted. New cstate: current_term: 2 leader_uuid: "d182e91f1aca4b21b897714bd8522dcc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d182e91f1aca4b21b897714bd8522dcc" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 40403 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:11:21.900079 27451 ts_tablet_manager.cc:1888] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 08:11:21.900223 27451 log.cc:1196] T a331a1e85ff94e5fa93b42583f1668d5 P 213ccbb05e6f44deb4a34d0a7041f7de: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/a331a1e85ff94e5fa93b42583f1668d5
I20240501 08:11:21.901016 24074 catalog_manager.cc:4830] TS 213ccbb05e6f44deb4a34d0a7041f7de (127.21.84.130:37857): tablet a331a1e85ff94e5fa93b42583f1668d5 (table rebalance_test_table_1 [id=60c95048bf784e6ab27a6c2db06a3f90]) successfully deleted
I20240501 08:11:21.901757 24208 tablet_service.cc:1514] Processing DeleteTablet for tablet 49cdeb66bf204db3ac2e6c32f0705382 with delete_type TABLET_DATA_TOMBSTONED (TS fb59098726224e7bae383f5c11820b39 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:46956
I20240501 08:11:21.902357 27452 tablet_replica.cc:330] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39: stopping tablet replica
I20240501 08:11:21.902526 27452 raft_consensus.cc:2229] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:21.902704 27452 raft_consensus.cc:2258] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:21.904119 27452 ts_tablet_manager.cc:1875] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:21.908411 27452 ts_tablet_manager.cc:1888] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 08:11:21.908540 27452 log.cc:1196] T 49cdeb66bf204db3ac2e6c32f0705382 P fb59098726224e7bae383f5c11820b39: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/49cdeb66bf204db3ac2e6c32f0705382
I20240501 08:11:21.909430 24073 catalog_manager.cc:4830] TS fb59098726224e7bae383f5c11820b39 (127.21.84.129:44759): tablet 49cdeb66bf204db3ac2e6c32f0705382 (table rebalance_test_table_4 [id=d5585afc570243fbbbb154585b168608]) successfully deleted
I20240501 08:11:22.188900 24339 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.200999 25562 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.202026 25822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.210614 25952 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.264771 27093 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.265202 25692 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.299268 25431 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.317318 24208 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.479534 25822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.482777 25562 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.486572 24339 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.493446 27093 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.502434 25692 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.502509 25952 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.545174 25431 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.558741 24208 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.828387 25822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.853302 24339 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.869393 25562 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.872146 25692 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.891369 27093 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.914746 25952 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.925632 25431 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.946029 24208 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
1ba60b54c62540a8a64e0fcd6b4eaa8f | 127.21.84.190:43065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.21.84.148:38087 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
1ad8fc587a734cc48f7f49b00fc0c0a1 | 127.21.84.134:45565 | HEALTHY | <none> | 2 | 0
213ccbb05e6f44deb4a34d0a7041f7de | 127.21.84.130:37857 | HEALTHY | <none> | 2 | 0
5ce52428c87b46369add4a8afaaef67c | 127.21.84.132:43389 | HEALTHY | <none> | 2 | 0
8a102eff724b4f459bc91203802847fb | 127.21.84.135:35569 | HEALTHY | <none> | 2 | 0
962160cac57040678e5cacd4b8b73c4b | 127.21.84.133:37341 | HEALTHY | <none> | 2 | 0
d182e91f1aca4b21b897714bd8522dcc | 127.21.84.136:40403 | HEALTHY | <none> | 2 | 0
f0c82b0e87d84b5eb3b2fd4f3e63ff1b | 127.21.84.131:35031 | HEALTHY | <none> | 2 | 0
fb59098726224e7bae383f5c11820b39 | 127.21.84.129:44759 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.21.84.129 | experimental | 127.21.84.129:44759
local_ip_for_outbound_sockets | 127.21.84.130 | experimental | 127.21.84.130:37857
local_ip_for_outbound_sockets | 127.21.84.131 | experimental | 127.21.84.131:35031
local_ip_for_outbound_sockets | 127.21.84.132 | experimental | 127.21.84.132:43389
local_ip_for_outbound_sockets | 127.21.84.133 | experimental | 127.21.84.133:37341
local_ip_for_outbound_sockets | 127.21.84.134 | experimental | 127.21.84.134:45565
local_ip_for_outbound_sockets | 127.21.84.135 | experimental | 127.21.84.135:35569
local_ip_for_outbound_sockets | 127.21.84.136 | experimental | 127.21.84.136:40403
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.21.84.129:44759
server_dump_info_path | /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.21.84.130:37857
server_dump_info_path | /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.21.84.131:35031
server_dump_info_path | /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.21.84.132:43389
server_dump_info_path | /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.21.84.133:37341
server_dump_info_path | /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.21.84.134:45565
server_dump_info_path | /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.21.84.135:35569
server_dump_info_path | /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.21.84.136:40403
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.21.84.148:38087 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 08:11:22.991631 21842 log_verifier.cc:126] Checking tablet 0705e5b48e84423fa252da5d62b3008b
I20240501 08:11:23.002589 21842 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 0705e5b48e84423fa252da5d62b3008b
I20240501 08:11:23.002704 21842 log_verifier.cc:126] Checking tablet 158f407899b6440cb33d57025ab56873
I20240501 08:11:23.012622 21842 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 158f407899b6440cb33d57025ab56873
I20240501 08:11:23.012728 21842 log_verifier.cc:126] Checking tablet 19eec94f19284c9e97efdbe449a5238d
I20240501 08:11:23.022536 21842 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 19eec94f19284c9e97efdbe449a5238d
I20240501 08:11:23.022630 21842 log_verifier.cc:126] Checking tablet 28f02b777fc8455893ccb4bd4d585952
I20240501 08:11:23.032270 21842 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 28f02b777fc8455893ccb4bd4d585952
I20240501 08:11:23.032430 21842 log_verifier.cc:126] Checking tablet 476e8260712e414ab298ea94d413044e
I20240501 08:11:23.045471 21842 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 476e8260712e414ab298ea94d413044e
I20240501 08:11:23.045563 21842 log_verifier.cc:126] Checking tablet 49cdeb66bf204db3ac2e6c32f0705382
I20240501 08:11:23.057988 21842 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 49cdeb66bf204db3ac2e6c32f0705382
I20240501 08:11:23.058109 21842 log_verifier.cc:126] Checking tablet 6305a4cfdfc3420594ff185a15723ad1
I20240501 08:11:23.067873 21842 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 6305a4cfdfc3420594ff185a15723ad1
I20240501 08:11:23.068001 21842 log_verifier.cc:126] Checking tablet 89a8cf2415454cb7b9157703f72913f4
I20240501 08:11:23.077792 21842 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 89a8cf2415454cb7b9157703f72913f4
I20240501 08:11:23.077911 21842 log_verifier.cc:126] Checking tablet 9b065f9074a74fd9945f22596e7594db
I20240501 08:11:23.087747 21842 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 9b065f9074a74fd9945f22596e7594db
I20240501 08:11:23.087853 21842 log_verifier.cc:126] Checking tablet a331a1e85ff94e5fa93b42583f1668d5
I20240501 08:11:23.099861 21842 log_verifier.cc:177] Verified matching terms for 7 ops in tablet a331a1e85ff94e5fa93b42583f1668d5
I20240501 08:11:23.099980 21842 log_verifier.cc:126] Checking tablet abc7faa250d14cecb417da206b1927a3
I20240501 08:11:23.109968 21842 log_verifier.cc:177] Verified matching terms for 7 ops in tablet abc7faa250d14cecb417da206b1927a3
I20240501 08:11:23.110074 21842 log_verifier.cc:126] Checking tablet b0c941c4d9ba4607b947765c218a1a60
I20240501 08:11:23.115374 21842 log_verifier.cc:177] Verified matching terms for 5 ops in tablet b0c941c4d9ba4607b947765c218a1a60
I20240501 08:11:23.115490 21842 log_verifier.cc:126] Checking tablet c3651f31063f43dd90576200ec3acbe5
I20240501 08:11:23.127107 21842 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c3651f31063f43dd90576200ec3acbe5
I20240501 08:11:23.127209 21842 log_verifier.cc:126] Checking tablet c4bf2462d8b949d6a2f734780fd0d746
I20240501 08:11:23.137167 21842 log_verifier.cc:177] Verified matching terms for 7 ops in tablet c4bf2462d8b949d6a2f734780fd0d746
I20240501 08:11:23.137277 21842 log_verifier.cc:126] Checking tablet c5b68189903e46e48fde14764c024bff
I20240501 08:11:23.146704 21842 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c5b68189903e46e48fde14764c024bff
I20240501 08:11:23.146804 21842 log_verifier.cc:126] Checking tablet dc01c06c62234c8e96be1da7795149c4
I20240501 08:11:23.156023 21842 log_verifier.cc:177] Verified matching terms for 3 ops in tablet dc01c06c62234c8e96be1da7795149c4
I20240501 08:11:23.156881 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 24146
I20240501 08:11:23.158200 24267 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:23.308822 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 24146
I20240501 08:11:23.342660 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 24277
I20240501 08:11:23.343782 24398 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:23.489346 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 24277
I20240501 08:11:23.513965 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 25369
I20240501 08:11:23.520269 25490 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:23.633785 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 25369
I20240501 08:11:23.653862 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 25500
I20240501 08:11:23.660151 25622 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:23.781543 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 25500
W20240501 08:11:23.801363 24074 connection.cc:535] server connection from 127.21.84.132:51143 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:11:23.801509 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 25630
I20240501 08:11:23.807137 25752 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:23.919631 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 25630
I20240501 08:11:23.939869 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 25760
I20240501 08:11:23.945431 25882 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:24.057992 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 25760
I20240501 08:11:24.078230 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 25890
I20240501 08:11:24.084124 26012 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:24.196470 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 25890
I20240501 08:11:24.216784 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 27030
I20240501 08:11:24.222617 27154 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:24.327180 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 27030
I20240501 08:11:24.345748 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 24056
I20240501 08:11:24.346856 24117 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:24.493683 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 24056
2024-05-01T08:11:24Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/1 (27733 ms)
[----------] 1 test from RebalanceParamTest (27733 ms total)
[----------] 1 test from DDLDuringRebalancingTest
[ RUN ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/1
I20240501 08:11:24.560750 21842 test_util.cc:268] Using random seed: 139997527
I20240501 08:11:24.562677 21842 ts_itest-base.cc:112] Starting cluster with:
I20240501 08:11:24.562744 21842 ts_itest-base.cc:113] --------------
I20240501 08:11:24.562789 21842 ts_itest-base.cc:114] 8 tablet servers
I20240501 08:11:24.562836 21842 ts_itest-base.cc:115] 3 replicas per TS
I20240501 08:11:24.562901 21842 ts_itest-base.cc:116] --------------
2024-05-01T08:11:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T08:11:24Z Disabled control of system clock
I20240501 08:11:24.591852 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.84.190:36935
--webserver_interface=127.21.84.190
--webserver_port=0
--builtin_ntp_servers=127.21.84.148:39605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.84.190:36935
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 08:11:24.917129 27589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:24.917472 27589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:24.917596 27589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:24.924976 27589 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:24.925138 27589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 08:11:24.925230 27589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:24.925307 27589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 08:11:24.925374 27589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 08:11:24.934605 27589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:39605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.21.84.190:36935
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.84.190:36935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.21.84.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:24.936038 27589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:24.937876 27589 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:24.949523 27595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:24.950163 27597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:24.951069 27594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:24.951956 27589 server_base.cc:1034] running on GCE node
I20240501 08:11:24.952720 27589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:24.954214 27589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:24.955447 27589 hybrid_clock.cc:648] HybridClock initialized: now 1714551084955386 us; error 53 us; skew 500 ppm
I20240501 08:11:24.955816 27589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:24.958092 27589 webserver.cc:460] Webserver started at http://127.21.84.190:42433/ using document root <none> and password file <none>
I20240501 08:11:24.958846 27589 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:24.958964 27589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:24.959267 27589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:24.962029 27589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "30c0f17c356e49fca6e933d80d164d83"
format_stamp: "Formatted at 2024-05-01 08:11:24 on dist-test-slave-9cgk"
I20240501 08:11:24.962874 27589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "30c0f17c356e49fca6e933d80d164d83"
format_stamp: "Formatted at 2024-05-01 08:11:24 on dist-test-slave-9cgk"
I20240501 08:11:24.968012 27589 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20240501 08:11:24.971343 27603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:24.972633 27589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 08:11:24.972820 27589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/wal
uuid: "30c0f17c356e49fca6e933d80d164d83"
format_stamp: "Formatted at 2024-05-01 08:11:24 on dist-test-slave-9cgk"
I20240501 08:11:24.973031 27589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:25.006358 27589 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:25.007246 27589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:25.007513 27589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:25.024891 27654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.190:36935 every 8 connection(s)
I20240501 08:11:25.024919 27589 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.190:36935
I20240501 08:11:25.026382 27589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 08:11:25.029911 27655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:25.031551 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 27589
I20240501 08:11:25.031755 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 08:11:25.041002 27655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Bootstrap starting.
I20240501 08:11:25.044441 27655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:25.045670 27655 log.cc:824] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:25.048257 27655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: No bootstrap required, opened a new log
I20240501 08:11:25.054275 27655 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30c0f17c356e49fca6e933d80d164d83" member_type: VOTER last_known_addr { host: "127.21.84.190" port: 36935 } }
I20240501 08:11:25.054602 27655 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:25.054679 27655 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c0f17c356e49fca6e933d80d164d83, State: Initialized, Role: FOLLOWER
I20240501 08:11:25.055202 27655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30c0f17c356e49fca6e933d80d164d83" member_type: VOTER last_known_addr { host: "127.21.84.190" port: 36935 } }
I20240501 08:11:25.055378 27655 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:25.055469 27655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:25.055586 27655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:25.057746 27655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30c0f17c356e49fca6e933d80d164d83" member_type: VOTER last_known_addr { host: "127.21.84.190" port: 36935 } }
I20240501 08:11:25.058259 27655 leader_election.cc:302] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30c0f17c356e49fca6e933d80d164d83; no voters:
I20240501 08:11:25.058789 27655 leader_election.cc:288] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:25.058945 27660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:25.059674 27660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83 [term 1 LEADER]: Becoming Leader. State: Replica: 30c0f17c356e49fca6e933d80d164d83, State: Running, Role: LEADER
I20240501 08:11:25.060398 27660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30c0f17c356e49fca6e933d80d164d83" member_type: VOTER last_known_addr { host: "127.21.84.190" port: 36935 } }
I20240501 08:11:25.060796 27655 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:11:25.063031 27662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30c0f17c356e49fca6e933d80d164d83. Latest consensus state: current_term: 1 leader_uuid: "30c0f17c356e49fca6e933d80d164d83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30c0f17c356e49fca6e933d80d164d83" member_type: VOTER last_known_addr { host: "127.21.84.190" port: 36935 } } }
I20240501 08:11:25.063349 27662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83 [sys.catalog]: This master's current role is: LEADER
I20240501 08:11:25.063844 27661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30c0f17c356e49fca6e933d80d164d83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30c0f17c356e49fca6e933d80d164d83" member_type: VOTER last_known_addr { host: "127.21.84.190" port: 36935 } } }
I20240501 08:11:25.064059 27661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83 [sys.catalog]: This master's current role is: LEADER
I20240501 08:11:25.064760 27668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:11:25.070745 27668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:11:25.079310 27668 catalog_manager.cc:1347] Generated new cluster ID: 18d62beff6fd439884cb3c0acdf3dd7d
I20240501 08:11:25.079452 27668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:11:25.114370 27668 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:11:25.116036 27668 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:11:25.127496 27668 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Generated new TSK 0
I20240501 08:11:25.128361 27668 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:11:25.137107 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.129:0
--local_ip_for_outbound_sockets=127.21.84.129
--webserver_interface=127.21.84.129
--webserver_port=0
--tserver_master_addrs=127.21.84.190:36935
--builtin_ntp_servers=127.21.84.148:39605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 08:11:25.477205 27679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:25.477595 27679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:25.477746 27679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:25.485139 27679 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:25.485270 27679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:25.485435 27679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.129
I20240501 08:11:25.494501 27679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:39605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.21.84.129
--webserver_port=0
--tserver_master_addrs=127.21.84.190:36935
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.129
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:25.495854 27679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:25.497537 27679 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:25.509640 27684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:25.510483 27687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:25.511078 27685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:25.511888 27679 server_base.cc:1034] running on GCE node
I20240501 08:11:25.512555 27679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:25.513509 27679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:25.514766 27679 hybrid_clock.cc:648] HybridClock initialized: now 1714551085514685 us; error 46 us; skew 500 ppm
I20240501 08:11:25.515116 27679 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:25.517441 27679 webserver.cc:460] Webserver started at http://127.21.84.129:33855/ using document root <none> and password file <none>
I20240501 08:11:25.518288 27679 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:25.518409 27679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:25.518725 27679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:25.521286 27679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
format_stamp: "Formatted at 2024-05-01 08:11:25 on dist-test-slave-9cgk"
I20240501 08:11:25.522097 27679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
format_stamp: "Formatted at 2024-05-01 08:11:25 on dist-test-slave-9cgk"
I20240501 08:11:25.527240 27679 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 08:11:25.530745 27693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:25.532172 27679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 08:11:25.532346 27679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
format_stamp: "Formatted at 2024-05-01 08:11:25 on dist-test-slave-9cgk"
I20240501 08:11:25.532550 27679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:25.551204 27679 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:25.552062 27679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:25.552357 27679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:25.553357 27679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:25.554949 27679 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:25.555047 27679 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:25.555171 27679 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:25.555241 27679 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:25.586751 27679 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.129:43471
I20240501 08:11:25.586807 27805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.129:43471 every 8 connection(s)
I20240501 08:11:25.588073 27679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 08:11:25.598212 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 27679
I20240501 08:11:25.598479 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 08:11:25.601047 27806 heartbeater.cc:347] Connected to a master server at 127.21.84.190:36935
I20240501 08:11:25.601392 27806 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:25.602478 27806 heartbeater.cc:511] Master 127.21.84.190:36935 requested a full tablet report, sending...
I20240501 08:11:25.602705 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.130:0
--local_ip_for_outbound_sockets=127.21.84.130
--webserver_interface=127.21.84.130
--webserver_port=0
--tserver_master_addrs=127.21.84.190:36935
--builtin_ntp_servers=127.21.84.148:39605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:11:25.604786 27619 ts_manager.cc:194] Registered new tserver with Master: 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:25.606324 27619 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.129:58153
W20240501 08:11:25.930523 27810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:25.930822 27810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:25.930918 27810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:25.938300 27810 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:25.938427 27810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:25.938549 27810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.130
I20240501 08:11:25.947618 27810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:39605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.21.84.130
--webserver_port=0
--tserver_master_addrs=127.21.84.190:36935
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.130
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:25.948910 27810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:25.950654 27810 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:25.962949 27816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:25.963481 27815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:25.964392 27818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:25.965272 27810 server_base.cc:1034] running on GCE node
I20240501 08:11:25.966065 27810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:25.967681 27810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:25.969041 27810 hybrid_clock.cc:648] HybridClock initialized: now 1714551085968819 us; error 67 us; skew 500 ppm
I20240501 08:11:25.969482 27810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:25.972556 27810 webserver.cc:460] Webserver started at http://127.21.84.130:42685/ using document root <none> and password file <none>
I20240501 08:11:25.973415 27810 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:25.973532 27810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:25.973851 27810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:25.977393 27810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "f1525cbc82a24d13b65f136695c903af"
format_stamp: "Formatted at 2024-05-01 08:11:25 on dist-test-slave-9cgk"
I20240501 08:11:25.978302 27810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "f1525cbc82a24d13b65f136695c903af"
format_stamp: "Formatted at 2024-05-01 08:11:25 on dist-test-slave-9cgk"
I20240501 08:11:25.985565 27810 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20240501 08:11:25.990505 27824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:25.992410 27810 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20240501 08:11:25.992609 27810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "f1525cbc82a24d13b65f136695c903af"
format_stamp: "Formatted at 2024-05-01 08:11:25 on dist-test-slave-9cgk"
I20240501 08:11:25.992871 27810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:26.041764 27810 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:26.042896 27810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:26.043180 27810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:26.044292 27810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:26.046100 27810 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:26.046211 27810 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:26.046290 27810 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:26.046341 27810 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:26.077839 27810 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.130:36503
I20240501 08:11:26.077893 27936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.130:36503 every 8 connection(s)
I20240501 08:11:26.079172 27810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 08:11:26.084022 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 27810
I20240501 08:11:26.084204 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 08:11:26.087580 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.131:0
--local_ip_for_outbound_sockets=127.21.84.131
--webserver_interface=127.21.84.131
--webserver_port=0
--tserver_master_addrs=127.21.84.190:36935
--builtin_ntp_servers=127.21.84.148:39605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:11:26.093077 27937 heartbeater.cc:347] Connected to a master server at 127.21.84.190:36935
I20240501 08:11:26.093425 27937 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:26.094285 27937 heartbeater.cc:511] Master 127.21.84.190:36935 requested a full tablet report, sending...
I20240501 08:11:26.095834 27619 ts_manager.cc:194] Registered new tserver with Master: f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:11:26.096566 27619 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.130:59611
W20240501 08:11:26.413446 27940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:26.413796 27940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:26.413969 27940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:26.421326 27940 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:26.421481 27940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:26.421651 27940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.131
I20240501 08:11:26.430613 27940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:39605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.21.84.131
--webserver_port=0
--tserver_master_addrs=127.21.84.190:36935
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.131
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:26.431970 27940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:26.433723 27940 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:26.445580 27945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:26.446421 27948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:26.447185 27946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:26.447990 27940 server_base.cc:1034] running on GCE node
I20240501 08:11:26.448616 27940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:26.449585 27940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:26.450786 27940 hybrid_clock.cc:648] HybridClock initialized: now 1714551086450728 us; error 57 us; skew 500 ppm
I20240501 08:11:26.451115 27940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:26.453408 27940 webserver.cc:460] Webserver started at http://127.21.84.131:34119/ using document root <none> and password file <none>
I20240501 08:11:26.454296 27940 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:26.454424 27940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:26.454751 27940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:26.457430 27940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "aad81047009a4ffcbc450b60030583b5"
format_stamp: "Formatted at 2024-05-01 08:11:26 on dist-test-slave-9cgk"
I20240501 08:11:26.458246 27940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "aad81047009a4ffcbc450b60030583b5"
format_stamp: "Formatted at 2024-05-01 08:11:26 on dist-test-slave-9cgk"
I20240501 08:11:26.463760 27940 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 08:11:26.467304 27954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:26.468777 27940 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 08:11:26.468964 27940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "aad81047009a4ffcbc450b60030583b5"
format_stamp: "Formatted at 2024-05-01 08:11:26 on dist-test-slave-9cgk"
I20240501 08:11:26.469187 27940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:26.499995 27940 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:26.500907 27940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:26.501175 27940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:26.502187 27940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:26.503688 27940 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:26.503783 27940 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:26.503897 27940 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:26.503968 27940 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:26.535128 27940 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.131:36305
I20240501 08:11:26.535181 28066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.131:36305 every 8 connection(s)
I20240501 08:11:26.536403 27940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 08:11:26.540740 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 27940
I20240501 08:11:26.540922 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 08:11:26.544126 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.132:0
--local_ip_for_outbound_sockets=127.21.84.132
--webserver_interface=127.21.84.132
--webserver_port=0
--tserver_master_addrs=127.21.84.190:36935
--builtin_ntp_servers=127.21.84.148:39605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:11:26.549237 28067 heartbeater.cc:347] Connected to a master server at 127.21.84.190:36935
I20240501 08:11:26.549543 28067 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:26.550408 28067 heartbeater.cc:511] Master 127.21.84.190:36935 requested a full tablet report, sending...
I20240501 08:11:26.551923 27619 ts_manager.cc:194] Registered new tserver with Master: aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:11:26.553056 27619 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.131:35507
I20240501 08:11:26.609309 27806 heartbeater.cc:503] Master 127.21.84.190:36935 was elected leader, sending a full tablet report...
W20240501 08:11:26.871589 28070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:26.871917 28070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:26.872005 28070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:26.879487 28070 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:26.879623 28070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:26.879739 28070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.132
I20240501 08:11:26.888407 28070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:39605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.21.84.132
--webserver_port=0
--tserver_master_addrs=127.21.84.190:36935
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.132
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:26.889707 28070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:26.891367 28070 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:26.903148 28075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:26.903892 28078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:26.904788 28076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:26.905586 28070 server_base.cc:1034] running on GCE node
I20240501 08:11:26.906226 28070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:26.907156 28070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:26.908370 28070 hybrid_clock.cc:648] HybridClock initialized: now 1714551086908330 us; error 38 us; skew 500 ppm
I20240501 08:11:26.908684 28070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:26.910939 28070 webserver.cc:460] Webserver started at http://127.21.84.132:34181/ using document root <none> and password file <none>
I20240501 08:11:26.911679 28070 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:26.911803 28070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:26.912107 28070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:26.914770 28070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
format_stamp: "Formatted at 2024-05-01 08:11:26 on dist-test-slave-9cgk"
I20240501 08:11:26.915529 28070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
format_stamp: "Formatted at 2024-05-01 08:11:26 on dist-test-slave-9cgk"
I20240501 08:11:26.920632 28070 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20240501 08:11:26.923805 28084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:26.925148 28070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:26.925309 28070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
format_stamp: "Formatted at 2024-05-01 08:11:26 on dist-test-slave-9cgk"
I20240501 08:11:26.925494 28070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:26.955471 28070 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:26.956350 28070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:26.956614 28070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:26.957635 28070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:26.959240 28070 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:26.959342 28070 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:26.959456 28070 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:26.959538 28070 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:26.989427 28070 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.132:41323
I20240501 08:11:26.989486 28196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.132:41323 every 8 connection(s)
I20240501 08:11:26.990782 28070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 08:11:26.995813 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 28070
I20240501 08:11:26.995971 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 08:11:26.999312 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.133:0
--local_ip_for_outbound_sockets=127.21.84.133
--webserver_interface=127.21.84.133
--webserver_port=0
--tserver_master_addrs=127.21.84.190:36935
--builtin_ntp_servers=127.21.84.148:39605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:11:27.004089 28197 heartbeater.cc:347] Connected to a master server at 127.21.84.190:36935
I20240501 08:11:27.004393 28197 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:27.005214 28197 heartbeater.cc:511] Master 127.21.84.190:36935 requested a full tablet report, sending...
I20240501 08:11:27.006722 27619 ts_manager.cc:194] Registered new tserver with Master: 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:11:27.007802 27619 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.132:34149
I20240501 08:11:27.099025 27937 heartbeater.cc:503] Master 127.21.84.190:36935 was elected leader, sending a full tablet report...
W20240501 08:11:27.326596 28200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:27.326867 28200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:27.326994 28200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:27.334513 28200 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:27.334637 28200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:27.334810 28200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.133
I20240501 08:11:27.343760 28200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:39605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.21.84.133
--webserver_port=0
--tserver_master_addrs=127.21.84.190:36935
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.133
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:27.345058 28200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:27.346731 28200 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:27.358711 28205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:27.359350 28206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:27.360322 28208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:27.361176 28200 server_base.cc:1034] running on GCE node
I20240501 08:11:27.361806 28200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:27.362738 28200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:27.363937 28200 hybrid_clock.cc:648] HybridClock initialized: now 1714551087363895 us; error 38 us; skew 500 ppm
I20240501 08:11:27.364248 28200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:27.366511 28200 webserver.cc:460] Webserver started at http://127.21.84.133:38715/ using document root <none> and password file <none>
I20240501 08:11:27.367230 28200 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:27.367334 28200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:27.367664 28200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:27.370362 28200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "31811708c6774037b82df44024b229b6"
format_stamp: "Formatted at 2024-05-01 08:11:27 on dist-test-slave-9cgk"
I20240501 08:11:27.371050 28200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "31811708c6774037b82df44024b229b6"
format_stamp: "Formatted at 2024-05-01 08:11:27 on dist-test-slave-9cgk"
I20240501 08:11:27.376075 28200 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 08:11:27.379354 28214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:27.380635 28200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 08:11:27.380786 28200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "31811708c6774037b82df44024b229b6"
format_stamp: "Formatted at 2024-05-01 08:11:27 on dist-test-slave-9cgk"
I20240501 08:11:27.380977 28200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:27.403498 28200 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:27.404336 28200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:27.404603 28200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:27.405521 28200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:27.406976 28200 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:27.407073 28200 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:27.407186 28200 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:27.407255 28200 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:27.437801 28200 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.133:38293
I20240501 08:11:27.437855 28326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.133:38293 every 8 connection(s)
I20240501 08:11:27.439321 28200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 08:11:27.442308 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 28200
I20240501 08:11:27.442492 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 08:11:27.445638 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.134:0
--local_ip_for_outbound_sockets=127.21.84.134
--webserver_interface=127.21.84.134
--webserver_port=0
--tserver_master_addrs=127.21.84.190:36935
--builtin_ntp_servers=127.21.84.148:39605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:11:27.452231 28327 heartbeater.cc:347] Connected to a master server at 127.21.84.190:36935
I20240501 08:11:27.452535 28327 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:27.453521 28327 heartbeater.cc:511] Master 127.21.84.190:36935 requested a full tablet report, sending...
I20240501 08:11:27.455088 27619 ts_manager.cc:194] Registered new tserver with Master: 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:11:27.455842 27619 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.133:43931
I20240501 08:11:27.555598 28067 heartbeater.cc:503] Master 127.21.84.190:36935 was elected leader, sending a full tablet report...
W20240501 08:11:27.771422 28330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:27.771711 28330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:27.771806 28330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:27.779062 28330 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:27.779178 28330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:27.779295 28330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.134
I20240501 08:11:27.787983 28330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:39605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.21.84.134
--webserver_port=0
--tserver_master_addrs=127.21.84.190:36935
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.134
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:27.789307 28330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:27.790998 28330 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:27.803150 28335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:27.804148 28338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:27.804697 28336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:27.805366 28330 server_base.cc:1034] running on GCE node
I20240501 08:11:27.805996 28330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:27.806995 28330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:27.808233 28330 hybrid_clock.cc:648] HybridClock initialized: now 1714551087808196 us; error 55 us; skew 500 ppm
I20240501 08:11:27.808593 28330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:27.810953 28330 webserver.cc:460] Webserver started at http://127.21.84.134:44003/ using document root <none> and password file <none>
I20240501 08:11:27.811738 28330 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:27.811856 28330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:27.812199 28330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:27.814851 28330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "9a022e1a501f4d44a0468054d6c3de47"
format_stamp: "Formatted at 2024-05-01 08:11:27 on dist-test-slave-9cgk"
I20240501 08:11:27.815614 28330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "9a022e1a501f4d44a0468054d6c3de47"
format_stamp: "Formatted at 2024-05-01 08:11:27 on dist-test-slave-9cgk"
I20240501 08:11:27.820750 28330 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240501 08:11:27.824208 28344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:27.825531 28330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:27.825696 28330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "9a022e1a501f4d44a0468054d6c3de47"
format_stamp: "Formatted at 2024-05-01 08:11:27 on dist-test-slave-9cgk"
I20240501 08:11:27.825883 28330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:27.840763 28330 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:27.841585 28330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:27.841838 28330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:27.842715 28330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:27.844094 28330 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:27.844190 28330 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:27.844301 28330 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:27.844372 28330 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:27.875703 28330 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.134:39933
I20240501 08:11:27.875747 28456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.134:39933 every 8 connection(s)
I20240501 08:11:27.877086 28330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 08:11:27.887430 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 28330
I20240501 08:11:27.887621 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 08:11:27.889784 28457 heartbeater.cc:347] Connected to a master server at 127.21.84.190:36935
I20240501 08:11:27.890897 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.135:0
--local_ip_for_outbound_sockets=127.21.84.135
--webserver_interface=127.21.84.135
--webserver_port=0
--tserver_master_addrs=127.21.84.190:36935
--builtin_ntp_servers=127.21.84.148:39605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:11:27.894201 28457 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:27.895123 28457 heartbeater.cc:511] Master 127.21.84.190:36935 requested a full tablet report, sending...
I20240501 08:11:27.896728 27619 ts_manager.cc:194] Registered new tserver with Master: 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:11:27.897500 27619 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.134:48005
I20240501 08:11:28.010906 28197 heartbeater.cc:503] Master 127.21.84.190:36935 was elected leader, sending a full tablet report...
W20240501 08:11:28.218942 28460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:28.219270 28460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:28.219411 28460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:28.226863 28460 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:28.227041 28460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:28.227217 28460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.135
I20240501 08:11:28.236776 28460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:39605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.21.84.135
--webserver_port=0
--tserver_master_addrs=127.21.84.190:36935
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.135
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:28.238214 28460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:28.239943 28460 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:28.252194 28465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:28.252772 28468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:28.253722 28466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:28.254491 28460 server_base.cc:1034] running on GCE node
I20240501 08:11:28.255091 28460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:28.256019 28460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:28.257210 28460 hybrid_clock.cc:648] HybridClock initialized: now 1714551088257160 us; error 41 us; skew 500 ppm
I20240501 08:11:28.257514 28460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:28.259841 28460 webserver.cc:460] Webserver started at http://127.21.84.135:42751/ using document root <none> and password file <none>
I20240501 08:11:28.260686 28460 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:28.260793 28460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:28.261135 28460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:28.263928 28460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "64998323a1f4482d8704bf20594d13a5"
format_stamp: "Formatted at 2024-05-01 08:11:28 on dist-test-slave-9cgk"
I20240501 08:11:28.264784 28460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "64998323a1f4482d8704bf20594d13a5"
format_stamp: "Formatted at 2024-05-01 08:11:28 on dist-test-slave-9cgk"
I20240501 08:11:28.270036 28460 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20240501 08:11:28.273320 28474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:28.274773 28460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:11:28.274911 28460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "64998323a1f4482d8704bf20594d13a5"
format_stamp: "Formatted at 2024-05-01 08:11:28 on dist-test-slave-9cgk"
I20240501 08:11:28.275084 28460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:28.293709 28460 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:28.294631 28460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:28.294854 28460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:28.295694 28460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:28.297077 28460 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:28.297160 28460 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:28.297237 28460 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:28.297307 28460 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:28.328477 28460 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.135:33503
I20240501 08:11:28.328533 28586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.135:33503 every 8 connection(s)
I20240501 08:11:28.329969 28460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 08:11:28.330906 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 28460
I20240501 08:11:28.331099 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 08:11:28.334314 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.136:0
--local_ip_for_outbound_sockets=127.21.84.136
--webserver_interface=127.21.84.136
--webserver_port=0
--tserver_master_addrs=127.21.84.190:36935
--builtin_ntp_servers=127.21.84.148:39605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:11:28.344220 28587 heartbeater.cc:347] Connected to a master server at 127.21.84.190:36935
I20240501 08:11:28.344584 28587 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:28.345674 28587 heartbeater.cc:511] Master 127.21.84.190:36935 requested a full tablet report, sending...
I20240501 08:11:28.347235 27619 ts_manager.cc:194] Registered new tserver with Master: 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:11:28.348012 27619 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.135:43571
I20240501 08:11:28.458791 28327 heartbeater.cc:503] Master 127.21.84.190:36935 was elected leader, sending a full tablet report...
W20240501 08:11:28.668383 28590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:28.668730 28590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:28.668877 28590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:28.676172 28590 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:11:28.676334 28590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:28.676492 28590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.136
I20240501 08:11:28.685426 28590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:39605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.21.84.136
--webserver_port=0
--tserver_master_addrs=127.21.84.190:36935
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.136
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:28.686813 28590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:28.688566 28590 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:28.700987 28596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:28.701745 28598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:28.702531 28595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:28.703269 28590 server_base.cc:1034] running on GCE node
I20240501 08:11:28.703868 28590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:28.704792 28590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:28.706022 28590 hybrid_clock.cc:648] HybridClock initialized: now 1714551088705963 us; error 56 us; skew 500 ppm
I20240501 08:11:28.706327 28590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:28.708628 28590 webserver.cc:460] Webserver started at http://127.21.84.136:45935/ using document root <none> and password file <none>
I20240501 08:11:28.709403 28590 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:28.709520 28590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:28.709828 28590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:28.712565 28590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "a29cd590999b4f32849ff5689cd53c11"
format_stamp: "Formatted at 2024-05-01 08:11:28 on dist-test-slave-9cgk"
I20240501 08:11:28.713351 28590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "a29cd590999b4f32849ff5689cd53c11"
format_stamp: "Formatted at 2024-05-01 08:11:28 on dist-test-slave-9cgk"
I20240501 08:11:28.718621 28590 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20240501 08:11:28.721889 28604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:28.723356 28590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 08:11:28.723555 28590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "a29cd590999b4f32849ff5689cd53c11"
format_stamp: "Formatted at 2024-05-01 08:11:28 on dist-test-slave-9cgk"
I20240501 08:11:28.723795 28590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:28.745617 28590 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:28.746585 28590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:28.746842 28590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:28.747804 28590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:28.749311 28590 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:28.749419 28590 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:28.749529 28590 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:28.749601 28590 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:28.779295 28590 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.136:46845
I20240501 08:11:28.779337 28716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.136:46845 every 8 connection(s)
I20240501 08:11:28.780797 28590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 08:11:28.787083 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 28590
I20240501 08:11:28.787223 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 08:11:28.793342 28717 heartbeater.cc:347] Connected to a master server at 127.21.84.190:36935
I20240501 08:11:28.793607 28717 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:28.794451 28717 heartbeater.cc:511] Master 127.21.84.190:36935 requested a full tablet report, sending...
I20240501 08:11:28.795742 27619 ts_manager.cc:194] Registered new tserver with Master: a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:11:28.796468 27619 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.136:48369
I20240501 08:11:28.803259 21842 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 08:11:28.819509 21842 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 08:11:28.819671 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 28200
I20240501 08:11:28.827391 28322 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:28.900326 28457 heartbeater.cc:503] Master 127.21.84.190:36935 was elected leader, sending a full tablet report...
I20240501 08:11:28.911511 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 28200
I20240501 08:11:28.927170 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 28330
I20240501 08:11:28.935148 28452 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:29.014525 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 28330
I20240501 08:11:29.033756 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 28460
I20240501 08:11:29.041818 28582 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:29.124747 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 28460
I20240501 08:11:29.145030 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 28590
I20240501 08:11:29.153385 28712 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:29.233651 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 28590
I20240501 08:11:33.001498 27619 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:33.065173 28001 tablet_service.cc:1467] Processing CreateTablet for tablet 6cea7a6a22b74bf48285c37ad52c9654 (DEFAULT_TABLE table=rebalance_test_table_0 [id=5f361e8ecdad4209a6f515928d7198eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:33.066532 28001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cea7a6a22b74bf48285c37ad52c9654. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:33.064710 28002 tablet_service.cc:1467] Processing CreateTablet for tablet cd1a890b5d1748b79cdf4bbac02bfecb (DEFAULT_TABLE table=rebalance_test_table_0 [id=5f361e8ecdad4209a6f515928d7198eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:33.068238 28002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd1a890b5d1748b79cdf4bbac02bfecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:33.068254 28131 tablet_service.cc:1467] Processing CreateTablet for tablet d3febc190f7445c197d8413c0ed82ece (DEFAULT_TABLE table=rebalance_test_table_0 [id=5f361e8ecdad4209a6f515928d7198eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:33.069321 28132 tablet_service.cc:1467] Processing CreateTablet for tablet cd1a890b5d1748b79cdf4bbac02bfecb (DEFAULT_TABLE table=rebalance_test_table_0 [id=5f361e8ecdad4209a6f515928d7198eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:33.069818 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3febc190f7445c197d8413c0ed82ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:33.070376 28132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd1a890b5d1748b79cdf4bbac02bfecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:33.077821 27871 tablet_service.cc:1467] Processing CreateTablet for tablet 6cea7a6a22b74bf48285c37ad52c9654 (DEFAULT_TABLE table=rebalance_test_table_0 [id=5f361e8ecdad4209a6f515928d7198eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:33.078811 27870 tablet_service.cc:1467] Processing CreateTablet for tablet d3febc190f7445c197d8413c0ed82ece (DEFAULT_TABLE table=rebalance_test_table_0 [id=5f361e8ecdad4209a6f515928d7198eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:33.080298 27870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3febc190f7445c197d8413c0ed82ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:33.079269 27872 tablet_service.cc:1467] Processing CreateTablet for tablet cd1a890b5d1748b79cdf4bbac02bfecb (DEFAULT_TABLE table=rebalance_test_table_0 [id=5f361e8ecdad4209a6f515928d7198eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:33.085582 27872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd1a890b5d1748b79cdf4bbac02bfecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:33.087348 27871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cea7a6a22b74bf48285c37ad52c9654. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:33.091648 28745 tablet_bootstrap.cc:492] T cd1a890b5d1748b79cdf4bbac02bfecb P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:33.095000 28745 tablet_bootstrap.cc:654] T cd1a890b5d1748b79cdf4bbac02bfecb P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:33.096148 28745 log.cc:824] T cd1a890b5d1748b79cdf4bbac02bfecb P 91d11eae5ef6432ca86eb8ed446b6ea6: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:33.096010 27740 tablet_service.cc:1467] Processing CreateTablet for tablet d3febc190f7445c197d8413c0ed82ece (DEFAULT_TABLE table=rebalance_test_table_0 [id=5f361e8ecdad4209a6f515928d7198eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:33.097126 27741 tablet_service.cc:1467] Processing CreateTablet for tablet 6cea7a6a22b74bf48285c37ad52c9654 (DEFAULT_TABLE table=rebalance_test_table_0 [id=5f361e8ecdad4209a6f515928d7198eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:33.098811 27740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3febc190f7445c197d8413c0ed82ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:33.098999 28745 tablet_bootstrap.cc:492] T cd1a890b5d1748b79cdf4bbac02bfecb P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:33.099381 28745 ts_tablet_manager.cc:1367] T cd1a890b5d1748b79cdf4bbac02bfecb P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240501 08:11:33.099625 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cea7a6a22b74bf48285c37ad52c9654. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:33.111155 28745 raft_consensus.cc:348] T cd1a890b5d1748b79cdf4bbac02bfecb P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:33.112118 28745 raft_consensus.cc:374] T cd1a890b5d1748b79cdf4bbac02bfecb P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:33.112324 28745 raft_consensus.cc:729] T cd1a890b5d1748b79cdf4bbac02bfecb P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:33.113487 28745 consensus_queue.cc:260] T cd1a890b5d1748b79cdf4bbac02bfecb P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:33.126164 28747 tablet_bootstrap.cc:492] T cd1a890b5d1748b79cdf4bbac02bfecb P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:33.126430 28748 tablet_bootstrap.cc:492] T 6cea7a6a22b74bf48285c37ad52c9654 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:33.127606 28745 ts_tablet_manager.cc:1398] T cd1a890b5d1748b79cdf4bbac02bfecb P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.028s user 0.013s sys 0.012s
I20240501 08:11:33.128087 28745 tablet_bootstrap.cc:492] T d3febc190f7445c197d8413c0ed82ece P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:33.129213 28750 tablet_bootstrap.cc:492] T d3febc190f7445c197d8413c0ed82ece P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:33.129734 28745 tablet_bootstrap.cc:654] T d3febc190f7445c197d8413c0ed82ece P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:33.132910 28750 tablet_bootstrap.cc:654] T d3febc190f7445c197d8413c0ed82ece P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:33.133211 28745 tablet_bootstrap.cc:492] T d3febc190f7445c197d8413c0ed82ece P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:33.133394 28745 ts_tablet_manager.cc:1367] T d3febc190f7445c197d8413c0ed82ece P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240501 08:11:33.133728 28748 tablet_bootstrap.cc:654] T 6cea7a6a22b74bf48285c37ad52c9654 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:33.134317 28747 tablet_bootstrap.cc:654] T cd1a890b5d1748b79cdf4bbac02bfecb P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:33.134413 28750 log.cc:824] T d3febc190f7445c197d8413c0ed82ece P 6331f77dae9d452184e8cd0cbf5f1b43: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:33.134936 28748 log.cc:824] T 6cea7a6a22b74bf48285c37ad52c9654 P aad81047009a4ffcbc450b60030583b5: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:33.135486 28745 raft_consensus.cc:348] T d3febc190f7445c197d8413c0ed82ece P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:33.136006 28745 raft_consensus.cc:374] T d3febc190f7445c197d8413c0ed82ece P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:33.136171 28745 raft_consensus.cc:729] T d3febc190f7445c197d8413c0ed82ece P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:33.136548 28745 consensus_queue.cc:260] T d3febc190f7445c197d8413c0ed82ece P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:33.136947 28747 log.cc:824] T cd1a890b5d1748b79cdf4bbac02bfecb P f1525cbc82a24d13b65f136695c903af: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:33.138388 28745 ts_tablet_manager.cc:1398] T d3febc190f7445c197d8413c0ed82ece P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:11:33.143564 28747 tablet_bootstrap.cc:492] T cd1a890b5d1748b79cdf4bbac02bfecb P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:33.143759 28750 tablet_bootstrap.cc:492] T d3febc190f7445c197d8413c0ed82ece P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:33.143846 28747 ts_tablet_manager.cc:1367] T cd1a890b5d1748b79cdf4bbac02bfecb P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.000s
I20240501 08:11:33.144059 28750 ts_tablet_manager.cc:1367] T d3febc190f7445c197d8413c0ed82ece P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20240501 08:11:33.144112 28748 tablet_bootstrap.cc:492] T 6cea7a6a22b74bf48285c37ad52c9654 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:33.144348 28748 ts_tablet_manager.cc:1367] T 6cea7a6a22b74bf48285c37ad52c9654 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.008s
I20240501 08:11:33.149762 28747 raft_consensus.cc:348] T cd1a890b5d1748b79cdf4bbac02bfecb P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:33.150189 28747 raft_consensus.cc:374] T cd1a890b5d1748b79cdf4bbac02bfecb P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:33.150314 28747 raft_consensus.cc:729] T cd1a890b5d1748b79cdf4bbac02bfecb P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:33.151301 28747 consensus_queue.cc:260] T cd1a890b5d1748b79cdf4bbac02bfecb P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:33.153214 28747 ts_tablet_manager.cc:1398] T cd1a890b5d1748b79cdf4bbac02bfecb P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.009s user 0.003s sys 0.006s
I20240501 08:11:33.153098 28748 raft_consensus.cc:348] T 6cea7a6a22b74bf48285c37ad52c9654 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:33.153105 28750 raft_consensus.cc:348] T d3febc190f7445c197d8413c0ed82ece P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:33.153507 28750 raft_consensus.cc:374] T d3febc190f7445c197d8413c0ed82ece P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:33.153548 28748 raft_consensus.cc:374] T 6cea7a6a22b74bf48285c37ad52c9654 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:33.153650 28750 raft_consensus.cc:729] T d3febc190f7445c197d8413c0ed82ece P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:33.153671 28748 raft_consensus.cc:729] T 6cea7a6a22b74bf48285c37ad52c9654 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:33.153795 28747 tablet_bootstrap.cc:492] T 6cea7a6a22b74bf48285c37ad52c9654 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:33.154358 28748 consensus_queue.cc:260] T 6cea7a6a22b74bf48285c37ad52c9654 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:33.154356 28750 consensus_queue.cc:260] T d3febc190f7445c197d8413c0ed82ece P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:33.155689 28747 tablet_bootstrap.cc:654] T 6cea7a6a22b74bf48285c37ad52c9654 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:33.159374 28747 tablet_bootstrap.cc:492] T 6cea7a6a22b74bf48285c37ad52c9654 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:33.159584 28747 ts_tablet_manager.cc:1367] T 6cea7a6a22b74bf48285c37ad52c9654 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 08:11:33.159983 28750 ts_tablet_manager.cc:1398] T d3febc190f7445c197d8413c0ed82ece P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20240501 08:11:33.160434 28750 tablet_bootstrap.cc:492] T 6cea7a6a22b74bf48285c37ad52c9654 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:33.161656 28748 ts_tablet_manager.cc:1398] T 6cea7a6a22b74bf48285c37ad52c9654 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.017s user 0.011s sys 0.006s
I20240501 08:11:33.162107 28748 tablet_bootstrap.cc:492] T cd1a890b5d1748b79cdf4bbac02bfecb P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:33.162281 28750 tablet_bootstrap.cc:654] T 6cea7a6a22b74bf48285c37ad52c9654 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:33.160851 28747 raft_consensus.cc:348] T 6cea7a6a22b74bf48285c37ad52c9654 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:33.164189 28748 tablet_bootstrap.cc:654] T cd1a890b5d1748b79cdf4bbac02bfecb P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:33.164386 28747 raft_consensus.cc:374] T 6cea7a6a22b74bf48285c37ad52c9654 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:33.164791 28747 raft_consensus.cc:729] T 6cea7a6a22b74bf48285c37ad52c9654 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:33.165144 28747 consensus_queue.cc:260] T 6cea7a6a22b74bf48285c37ad52c9654 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:33.165704 28747 ts_tablet_manager.cc:1398] T 6cea7a6a22b74bf48285c37ad52c9654 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:11:33.166201 28747 tablet_bootstrap.cc:492] T d3febc190f7445c197d8413c0ed82ece P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:33.166662 28750 tablet_bootstrap.cc:492] T 6cea7a6a22b74bf48285c37ad52c9654 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:33.166836 28750 ts_tablet_manager.cc:1367] T 6cea7a6a22b74bf48285c37ad52c9654 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.005s
I20240501 08:11:33.167863 28750 raft_consensus.cc:348] T 6cea7a6a22b74bf48285c37ad52c9654 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:33.168130 28750 raft_consensus.cc:374] T 6cea7a6a22b74bf48285c37ad52c9654 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:33.168242 28750 raft_consensus.cc:729] T 6cea7a6a22b74bf48285c37ad52c9654 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:33.168495 28750 consensus_queue.cc:260] T 6cea7a6a22b74bf48285c37ad52c9654 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:33.168918 28747 tablet_bootstrap.cc:654] T d3febc190f7445c197d8413c0ed82ece P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:33.169064 28750 ts_tablet_manager.cc:1398] T 6cea7a6a22b74bf48285c37ad52c9654 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:11:33.172257 28748 tablet_bootstrap.cc:492] T cd1a890b5d1748b79cdf4bbac02bfecb P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:33.172411 28748 ts_tablet_manager.cc:1367] T cd1a890b5d1748b79cdf4bbac02bfecb P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.007s
I20240501 08:11:33.173053 28747 tablet_bootstrap.cc:492] T d3febc190f7445c197d8413c0ed82ece P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:33.173218 28747 ts_tablet_manager.cc:1367] T d3febc190f7445c197d8413c0ed82ece P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 08:11:33.173533 28748 raft_consensus.cc:348] T cd1a890b5d1748b79cdf4bbac02bfecb P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:33.173735 28748 raft_consensus.cc:374] T cd1a890b5d1748b79cdf4bbac02bfecb P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:33.173831 28748 raft_consensus.cc:729] T cd1a890b5d1748b79cdf4bbac02bfecb P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:33.173924 28747 raft_consensus.cc:348] T d3febc190f7445c197d8413c0ed82ece P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:33.174166 28747 raft_consensus.cc:374] T d3febc190f7445c197d8413c0ed82ece P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:33.174083 28748 consensus_queue.cc:260] T cd1a890b5d1748b79cdf4bbac02bfecb P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:33.174240 28747 raft_consensus.cc:729] T d3febc190f7445c197d8413c0ed82ece P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:33.174659 28748 ts_tablet_manager.cc:1398] T cd1a890b5d1748b79cdf4bbac02bfecb P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:11:33.174535 28747 consensus_queue.cc:260] T d3febc190f7445c197d8413c0ed82ece P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:33.175083 28747 ts_tablet_manager.cc:1398] T d3febc190f7445c197d8413c0ed82ece P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:33.214655 28756 raft_consensus.cc:482] T d3febc190f7445c197d8413c0ed82ece P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:33.214929 28756 raft_consensus.cc:504] T d3febc190f7445c197d8413c0ed82ece P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:33.216624 28756 leader_election.cc:288] T d3febc190f7445c197d8413c0ed82ece P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:11:33.223574 28152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3febc190f7445c197d8413c0ed82ece" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:11:33.223575 27892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3febc190f7445c197d8413c0ed82ece" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:11:33.224038 28152 raft_consensus.cc:2454] T d3febc190f7445c197d8413c0ed82ece P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:11:33.224047 27892 raft_consensus.cc:2454] T d3febc190f7445c197d8413c0ed82ece P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:11:33.224635 27697 leader_election.cc:302] T d3febc190f7445c197d8413c0ed82ece P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:33.225023 28756 raft_consensus.cc:2781] T d3febc190f7445c197d8413c0ed82ece P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:33.225163 28756 raft_consensus.cc:482] T d3febc190f7445c197d8413c0ed82ece P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:33.225262 28756 raft_consensus.cc:3037] T d3febc190f7445c197d8413c0ed82ece P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:33.227272 28756 raft_consensus.cc:504] T d3febc190f7445c197d8413c0ed82ece P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:33.227931 28756 leader_election.cc:288] T d3febc190f7445c197d8413c0ed82ece P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Requested vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:11:33.228381 27892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3febc190f7445c197d8413c0ed82ece" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:11:33.228432 28152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3febc190f7445c197d8413c0ed82ece" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:11:33.228602 28152 raft_consensus.cc:3037] T d3febc190f7445c197d8413c0ed82ece P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:33.228598 27892 raft_consensus.cc:3037] T d3febc190f7445c197d8413c0ed82ece P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:33.231738 27892 raft_consensus.cc:2454] T d3febc190f7445c197d8413c0ed82ece P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:11:33.231738 28152 raft_consensus.cc:2454] T d3febc190f7445c197d8413c0ed82ece P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:11:33.232254 27694 leader_election.cc:302] T d3febc190f7445c197d8413c0ed82ece P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:33.232616 28756 raft_consensus.cc:2781] T d3febc190f7445c197d8413c0ed82ece P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:33.233036 28756 raft_consensus.cc:686] T d3febc190f7445c197d8413c0ed82ece P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:11:33.233551 28756 consensus_queue.cc:237] T d3febc190f7445c197d8413c0ed82ece P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:33.239012 27619 catalog_manager.cc:5484] T d3febc190f7445c197d8413c0ed82ece P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } }
W20240501 08:11:33.247041 28198 tablet.cc:2369] T cd1a890b5d1748b79cdf4bbac02bfecb P 91d11eae5ef6432ca86eb8ed446b6ea6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:11:33.275806 28756 raft_consensus.cc:482] T 6cea7a6a22b74bf48285c37ad52c9654 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:33.276072 28756 raft_consensus.cc:504] T 6cea7a6a22b74bf48285c37ad52c9654 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:33.276801 28756 leader_election.cc:288] T 6cea7a6a22b74bf48285c37ad52c9654 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:11:33.277341 27892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cea7a6a22b74bf48285c37ad52c9654" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:11:33.277626 27892 raft_consensus.cc:2454] T 6cea7a6a22b74bf48285c37ad52c9654 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:11:33.278049 27697 leader_election.cc:302] T 6cea7a6a22b74bf48285c37ad52c9654 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:33.278368 28756 raft_consensus.cc:2781] T 6cea7a6a22b74bf48285c37ad52c9654 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:33.278479 28756 raft_consensus.cc:482] T 6cea7a6a22b74bf48285c37ad52c9654 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:33.278591 28756 raft_consensus.cc:3037] T 6cea7a6a22b74bf48285c37ad52c9654 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:33.281050 28756 raft_consensus.cc:504] T 6cea7a6a22b74bf48285c37ad52c9654 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:33.281803 28756 leader_election.cc:288] T 6cea7a6a22b74bf48285c37ad52c9654 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Requested vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:11:33.282277 27892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cea7a6a22b74bf48285c37ad52c9654" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:11:33.282483 27892 raft_consensus.cc:3037] T 6cea7a6a22b74bf48285c37ad52c9654 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:33.283459 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cea7a6a22b74bf48285c37ad52c9654" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:33.283519 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cea7a6a22b74bf48285c37ad52c9654" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:33.283962 28022 raft_consensus.cc:2454] T 6cea7a6a22b74bf48285c37ad52c9654 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:11:33.284814 27892 raft_consensus.cc:2454] T 6cea7a6a22b74bf48285c37ad52c9654 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:11:33.285212 27697 leader_election.cc:302] T 6cea7a6a22b74bf48285c37ad52c9654 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, f1525cbc82a24d13b65f136695c903af; no voters: aad81047009a4ffcbc450b60030583b5
I20240501 08:11:33.285480 28756 raft_consensus.cc:2781] T 6cea7a6a22b74bf48285c37ad52c9654 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:33.285683 28756 raft_consensus.cc:686] T 6cea7a6a22b74bf48285c37ad52c9654 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:11:33.285964 28756 consensus_queue.cc:237] T 6cea7a6a22b74bf48285c37ad52c9654 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:33.288594 27619 catalog_manager.cc:5484] T 6cea7a6a22b74bf48285c37ad52c9654 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } }
W20240501 08:11:33.293350 28068 tablet.cc:2369] T 6cea7a6a22b74bf48285c37ad52c9654 P aad81047009a4ffcbc450b60030583b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:11:33.326745 28749 raft_consensus.cc:482] T cd1a890b5d1748b79cdf4bbac02bfecb P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:33.326936 28749 raft_consensus.cc:504] T cd1a890b5d1748b79cdf4bbac02bfecb P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:33.328570 28749 leader_election.cc:288] T cd1a890b5d1748b79cdf4bbac02bfecb P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:11:33.336947 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd1a890b5d1748b79cdf4bbac02bfecb" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
W20240501 08:11:33.337067 27938 tablet.cc:2369] T 6cea7a6a22b74bf48285c37ad52c9654 P f1525cbc82a24d13b65f136695c903af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:11:33.337034 27892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd1a890b5d1748b79cdf4bbac02bfecb" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:11:33.337255 28021 raft_consensus.cc:2454] T cd1a890b5d1748b79cdf4bbac02bfecb P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:33.337275 27892 raft_consensus.cc:2454] T cd1a890b5d1748b79cdf4bbac02bfecb P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:33.337796 28088 leader_election.cc:302] T cd1a890b5d1748b79cdf4bbac02bfecb P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:33.338249 28749 raft_consensus.cc:2781] T cd1a890b5d1748b79cdf4bbac02bfecb P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:33.338375 28749 raft_consensus.cc:482] T cd1a890b5d1748b79cdf4bbac02bfecb P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:33.338467 28749 raft_consensus.cc:3037] T cd1a890b5d1748b79cdf4bbac02bfecb P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:33.340478 28749 raft_consensus.cc:504] T cd1a890b5d1748b79cdf4bbac02bfecb P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:33.341176 28749 leader_election.cc:288] T cd1a890b5d1748b79cdf4bbac02bfecb P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:11:33.341606 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd1a890b5d1748b79cdf4bbac02bfecb" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:33.341702 27892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd1a890b5d1748b79cdf4bbac02bfecb" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:11:33.341886 27892 raft_consensus.cc:3037] T cd1a890b5d1748b79cdf4bbac02bfecb P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:33.341876 28021 raft_consensus.cc:3037] T cd1a890b5d1748b79cdf4bbac02bfecb P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:33.344743 27892 raft_consensus.cc:2454] T cd1a890b5d1748b79cdf4bbac02bfecb P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:33.344787 28021 raft_consensus.cc:2454] T cd1a890b5d1748b79cdf4bbac02bfecb P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:33.345249 28087 leader_election.cc:302] T cd1a890b5d1748b79cdf4bbac02bfecb P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:33.345588 28749 raft_consensus.cc:2781] T cd1a890b5d1748b79cdf4bbac02bfecb P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
W20240501 08:11:33.345657 27807 tablet.cc:2369] T 6cea7a6a22b74bf48285c37ad52c9654 P 6331f77dae9d452184e8cd0cbf5f1b43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:11:33.346107 28749 raft_consensus.cc:686] T cd1a890b5d1748b79cdf4bbac02bfecb P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:11:33.346658 28749 consensus_queue.cc:237] T cd1a890b5d1748b79cdf4bbac02bfecb P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:33.350948 27619 catalog_manager.cc:5484] T cd1a890b5d1748b79cdf4bbac02bfecb P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } }
W20240501 08:11:33.697211 28771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:33.697546 28771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:33.705013 28771 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 08:11:33.767028 28756 consensus_queue.cc:1035] T d3febc190f7445c197d8413c0ed82ece P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:33.782541 28021 raft_consensus.cc:3037] T 6cea7a6a22b74bf48285c37ad52c9654 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:33.786502 28756 consensus_queue.cc:1035] T d3febc190f7445c197d8413c0ed82ece P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:33.795739 28756 consensus_queue.cc:1035] T 6cea7a6a22b74bf48285c37ad52c9654 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:33.828348 28749 consensus_queue.cc:1035] T cd1a890b5d1748b79cdf4bbac02bfecb P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:33.862226 28756 consensus_queue.cc:1035] T 6cea7a6a22b74bf48285c37ad52c9654 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:33.863132 28749 consensus_queue.cc:1035] T cd1a890b5d1748b79cdf4bbac02bfecb P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 303.655 ms
time per row: 0.151827 ms
I20240501 08:11:34.162518 27614 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:34.171859 27740 tablet_service.cc:1467] Processing CreateTablet for tablet 132d1b9866574dcea3beed5835e4a22f (DEFAULT_TABLE table=rebalance_test_table_1 [id=0ee45bbb25734f82b00abd2da25300d1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:34.172381 27740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 132d1b9866574dcea3beed5835e4a22f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:34.172770 28132 tablet_service.cc:1467] Processing CreateTablet for tablet 132d1b9866574dcea3beed5835e4a22f (DEFAULT_TABLE table=rebalance_test_table_1 [id=0ee45bbb25734f82b00abd2da25300d1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:34.172801 27872 tablet_service.cc:1467] Processing CreateTablet for tablet 7f7211c8c0e041bf89930aa50a4438ea (DEFAULT_TABLE table=rebalance_test_table_1 [id=0ee45bbb25734f82b00abd2da25300d1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:34.173199 27872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f7211c8c0e041bf89930aa50a4438ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:34.173221 28132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 132d1b9866574dcea3beed5835e4a22f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:34.178303 28131 tablet_service.cc:1467] Processing CreateTablet for tablet 30130709805e4fa6ae94a46034710863 (DEFAULT_TABLE table=rebalance_test_table_1 [id=0ee45bbb25734f82b00abd2da25300d1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:34.178721 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30130709805e4fa6ae94a46034710863. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:34.179548 28843 tablet_bootstrap.cc:492] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:34.181116 28843 tablet_bootstrap.cc:654] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:34.183722 27739 tablet_service.cc:1467] Processing CreateTablet for tablet 7f7211c8c0e041bf89930aa50a4438ea (DEFAULT_TABLE table=rebalance_test_table_1 [id=0ee45bbb25734f82b00abd2da25300d1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:34.183900 28002 tablet_service.cc:1467] Processing CreateTablet for tablet 30130709805e4fa6ae94a46034710863 (DEFAULT_TABLE table=rebalance_test_table_1 [id=0ee45bbb25734f82b00abd2da25300d1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:34.184012 28001 tablet_service.cc:1467] Processing CreateTablet for tablet 132d1b9866574dcea3beed5835e4a22f (DEFAULT_TABLE table=rebalance_test_table_1 [id=0ee45bbb25734f82b00abd2da25300d1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:34.184317 27739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f7211c8c0e041bf89930aa50a4438ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:34.184337 28002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30130709805e4fa6ae94a46034710863. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:34.184779 28000 tablet_service.cc:1467] Processing CreateTablet for tablet 7f7211c8c0e041bf89930aa50a4438ea (DEFAULT_TABLE table=rebalance_test_table_1 [id=0ee45bbb25734f82b00abd2da25300d1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:34.185262 28000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f7211c8c0e041bf89930aa50a4438ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:34.190004 27741 tablet_service.cc:1467] Processing CreateTablet for tablet 30130709805e4fa6ae94a46034710863 (DEFAULT_TABLE table=rebalance_test_table_1 [id=0ee45bbb25734f82b00abd2da25300d1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:34.190444 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30130709805e4fa6ae94a46034710863. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:34.194154 28001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 132d1b9866574dcea3beed5835e4a22f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:34.197953 28843 tablet_bootstrap.cc:492] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:34.198138 28843 ts_tablet_manager.cc:1367] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.000s
I20240501 08:11:34.198180 28844 tablet_bootstrap.cc:492] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:34.198783 28847 tablet_bootstrap.cc:492] T 30130709805e4fa6ae94a46034710863 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:34.199028 28843 raft_consensus.cc:348] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:34.199239 28843 raft_consensus.cc:374] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:34.199328 28843 raft_consensus.cc:729] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:34.199548 28843 consensus_queue.cc:260] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:34.200778 28843 ts_tablet_manager.cc:1398] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:34.201694 28847 tablet_bootstrap.cc:654] T 30130709805e4fa6ae94a46034710863 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:34.206588 28846 tablet_bootstrap.cc:492] T 132d1b9866574dcea3beed5835e4a22f P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:34.208168 28844 tablet_bootstrap.cc:654] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:34.208635 28846 tablet_bootstrap.cc:654] T 132d1b9866574dcea3beed5835e4a22f P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:34.208959 28847 tablet_bootstrap.cc:492] T 30130709805e4fa6ae94a46034710863 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:34.209146 28847 ts_tablet_manager.cc:1367] T 30130709805e4fa6ae94a46034710863 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240501 08:11:34.210052 28847 raft_consensus.cc:348] T 30130709805e4fa6ae94a46034710863 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:34.210232 28847 raft_consensus.cc:374] T 30130709805e4fa6ae94a46034710863 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:34.210318 28847 raft_consensus.cc:729] T 30130709805e4fa6ae94a46034710863 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:34.210580 28847 consensus_queue.cc:260] T 30130709805e4fa6ae94a46034710863 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:34.210680 28844 tablet_bootstrap.cc:492] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:34.210863 28844 ts_tablet_manager.cc:1367] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240501 08:11:34.211078 28847 ts_tablet_manager.cc:1398] T 30130709805e4fa6ae94a46034710863 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:11:34.211306 28847 tablet_bootstrap.cc:492] T 132d1b9866574dcea3beed5835e4a22f P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:34.211653 28844 raft_consensus.cc:348] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:34.212263 28844 raft_consensus.cc:374] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:34.212379 28844 raft_consensus.cc:729] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:34.212813 28847 tablet_bootstrap.cc:654] T 132d1b9866574dcea3beed5835e4a22f P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:34.212705 28844 consensus_queue.cc:260] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:34.213676 28844 ts_tablet_manager.cc:1398] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240501 08:11:34.214030 28844 tablet_bootstrap.cc:492] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:34.216183 28844 tablet_bootstrap.cc:654] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:34.218122 28844 tablet_bootstrap.cc:492] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:34.218281 28844 ts_tablet_manager.cc:1367] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 08:11:34.218542 28846 tablet_bootstrap.cc:492] T 132d1b9866574dcea3beed5835e4a22f P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:34.218730 28846 ts_tablet_manager.cc:1367] T 132d1b9866574dcea3beed5835e4a22f P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.010s
I20240501 08:11:34.218988 28844 raft_consensus.cc:348] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:34.219220 28844 raft_consensus.cc:374] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:34.219297 28844 raft_consensus.cc:729] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:34.219549 28844 consensus_queue.cc:260] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:34.220093 28844 ts_tablet_manager.cc:1398] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:34.220120 28846 raft_consensus.cc:348] T 132d1b9866574dcea3beed5835e4a22f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:34.220347 28846 raft_consensus.cc:374] T 132d1b9866574dcea3beed5835e4a22f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:34.220445 28846 raft_consensus.cc:729] T 132d1b9866574dcea3beed5835e4a22f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:34.220742 28846 consensus_queue.cc:260] T 132d1b9866574dcea3beed5835e4a22f P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:34.221225 28847 tablet_bootstrap.cc:492] T 132d1b9866574dcea3beed5835e4a22f P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:34.221325 28846 ts_tablet_manager.cc:1398] T 132d1b9866574dcea3beed5835e4a22f P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:11:34.221403 28847 ts_tablet_manager.cc:1367] T 132d1b9866574dcea3beed5835e4a22f P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.001s
I20240501 08:11:34.222208 28847 raft_consensus.cc:348] T 132d1b9866574dcea3beed5835e4a22f P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:34.222399 28847 raft_consensus.cc:374] T 132d1b9866574dcea3beed5835e4a22f P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:34.222463 28846 tablet_bootstrap.cc:492] T 7f7211c8c0e041bf89930aa50a4438ea P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:34.222473 28847 raft_consensus.cc:729] T 132d1b9866574dcea3beed5835e4a22f P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:34.222719 28847 consensus_queue.cc:260] T 132d1b9866574dcea3beed5835e4a22f P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:34.223102 28847 ts_tablet_manager.cc:1398] T 132d1b9866574dcea3beed5835e4a22f P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:11:34.223446 28847 tablet_bootstrap.cc:492] T 7f7211c8c0e041bf89930aa50a4438ea P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:34.224378 28846 tablet_bootstrap.cc:654] T 7f7211c8c0e041bf89930aa50a4438ea P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:34.224836 28847 tablet_bootstrap.cc:654] T 7f7211c8c0e041bf89930aa50a4438ea P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:34.226588 28846 tablet_bootstrap.cc:492] T 7f7211c8c0e041bf89930aa50a4438ea P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:34.226761 28846 ts_tablet_manager.cc:1367] T 7f7211c8c0e041bf89930aa50a4438ea P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 08:11:34.227525 28846 raft_consensus.cc:348] T 7f7211c8c0e041bf89930aa50a4438ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:34.227777 28846 raft_consensus.cc:374] T 7f7211c8c0e041bf89930aa50a4438ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:34.227869 28846 raft_consensus.cc:729] T 7f7211c8c0e041bf89930aa50a4438ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:34.228122 28846 consensus_queue.cc:260] T 7f7211c8c0e041bf89930aa50a4438ea P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:34.228598 28846 ts_tablet_manager.cc:1398] T 7f7211c8c0e041bf89930aa50a4438ea P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:11:34.228941 28846 tablet_bootstrap.cc:492] T 30130709805e4fa6ae94a46034710863 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:34.230588 28846 tablet_bootstrap.cc:654] T 30130709805e4fa6ae94a46034710863 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:34.232082 28847 tablet_bootstrap.cc:492] T 7f7211c8c0e041bf89930aa50a4438ea P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:34.232261 28847 ts_tablet_manager.cc:1367] T 7f7211c8c0e041bf89930aa50a4438ea P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 08:11:34.233237 28846 tablet_bootstrap.cc:492] T 30130709805e4fa6ae94a46034710863 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:34.233103 28847 raft_consensus.cc:348] T 7f7211c8c0e041bf89930aa50a4438ea P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:34.233314 28847 raft_consensus.cc:374] T 7f7211c8c0e041bf89930aa50a4438ea P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:34.233371 28846 ts_tablet_manager.cc:1367] T 30130709805e4fa6ae94a46034710863 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240501 08:11:34.233410 28847 raft_consensus.cc:729] T 7f7211c8c0e041bf89930aa50a4438ea P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:34.233660 28847 consensus_queue.cc:260] T 7f7211c8c0e041bf89930aa50a4438ea P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:34.234249 28847 ts_tablet_manager.cc:1398] T 7f7211c8c0e041bf89930aa50a4438ea P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:11:34.234155 28846 raft_consensus.cc:348] T 30130709805e4fa6ae94a46034710863 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:34.234336 28846 raft_consensus.cc:374] T 30130709805e4fa6ae94a46034710863 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:34.234426 28846 raft_consensus.cc:729] T 30130709805e4fa6ae94a46034710863 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:34.234654 28846 consensus_queue.cc:260] T 30130709805e4fa6ae94a46034710863 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:34.235116 28846 ts_tablet_manager.cc:1398] T 30130709805e4fa6ae94a46034710863 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:34.330205 28749 raft_consensus.cc:482] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:34.330440 28749 raft_consensus.cc:504] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:34.331133 28749 leader_election.cc:288] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:11:34.332373 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30130709805e4fa6ae94a46034710863" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:34.332609 28022 raft_consensus.cc:2454] T 30130709805e4fa6ae94a46034710863 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:34.333060 28087 leader_election.cc:302] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:34.333340 28749 raft_consensus.cc:2781] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:34.333427 28749 raft_consensus.cc:482] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:34.333514 28749 raft_consensus.cc:3037] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:34.335492 28749 raft_consensus.cc:504] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:34.336252 28749 leader_election.cc:288] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:11:34.336773 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30130709805e4fa6ae94a46034710863" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:34.336970 28022 raft_consensus.cc:3037] T 30130709805e4fa6ae94a46034710863 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:34.337818 27760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30130709805e4fa6ae94a46034710863" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:11:34.337863 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30130709805e4fa6ae94a46034710863" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:11:34.338268 27760 raft_consensus.cc:3037] T 30130709805e4fa6ae94a46034710863 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:34.339685 28022 raft_consensus.cc:2454] T 30130709805e4fa6ae94a46034710863 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:34.340181 28087 leader_election.cc:302] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:34.340576 28749 raft_consensus.cc:2781] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:34.340795 27760 raft_consensus.cc:2454] T 30130709805e4fa6ae94a46034710863 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:34.340795 28749 raft_consensus.cc:686] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:11:34.341435 28749 consensus_queue.cc:237] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:34.344166 27614 catalog_manager.cc:5484] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:34.367453 28797 raft_consensus.cc:482] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:34.367652 28797 raft_consensus.cc:504] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:34.369208 28797 leader_election.cc:288] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:11:34.376938 27760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f7211c8c0e041bf89930aa50a4438ea" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:11:34.377257 27760 raft_consensus.cc:2454] T 7f7211c8c0e041bf89930aa50a4438ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:11:34.377342 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f7211c8c0e041bf89930aa50a4438ea" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:34.377615 28022 raft_consensus.cc:2454] T 7f7211c8c0e041bf89930aa50a4438ea P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:11:34.377789 27827 leader_election.cc:302] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:34.378276 28797 raft_consensus.cc:2781] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:34.378408 28797 raft_consensus.cc:482] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:34.378508 28797 raft_consensus.cc:3037] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:34.380493 28797 raft_consensus.cc:504] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:34.381080 28797 leader_election.cc:288] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Requested vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:11:34.381531 27760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f7211c8c0e041bf89930aa50a4438ea" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:11:34.381579 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f7211c8c0e041bf89930aa50a4438ea" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:34.381742 27760 raft_consensus.cc:3037] T 7f7211c8c0e041bf89930aa50a4438ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:34.381858 28022 raft_consensus.cc:3037] T 7f7211c8c0e041bf89930aa50a4438ea P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:34.383677 27760 raft_consensus.cc:2454] T 7f7211c8c0e041bf89930aa50a4438ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:11:34.383666 28022 raft_consensus.cc:2454] T 7f7211c8c0e041bf89930aa50a4438ea P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:11:34.384083 27827 leader_election.cc:302] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:34.384373 28797 raft_consensus.cc:2781] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:34.384794 28797 raft_consensus.cc:686] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Becoming Leader. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Running, Role: LEADER
I20240501 08:11:34.385234 28797 consensus_queue.cc:237] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:34.388954 27614 catalog_manager.cc:5484] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af reported cstate change: term changed from 0 to 1, leader changed from <none> to f1525cbc82a24d13b65f136695c903af (127.21.84.130). New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:34.409351 28749 raft_consensus.cc:482] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:34.409535 28749 raft_consensus.cc:504] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:34.410182 28749 leader_election.cc:288] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:11:34.410867 27760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "132d1b9866574dcea3beed5835e4a22f" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:11:34.410867 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "132d1b9866574dcea3beed5835e4a22f" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:34.411118 28022 raft_consensus.cc:2454] T 132d1b9866574dcea3beed5835e4a22f P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:34.411118 27760 raft_consensus.cc:2454] T 132d1b9866574dcea3beed5835e4a22f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:34.411521 28087 leader_election.cc:302] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:34.411762 28749 raft_consensus.cc:2781] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:34.411886 28749 raft_consensus.cc:482] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:34.411972 28749 raft_consensus.cc:3037] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:34.413887 28749 raft_consensus.cc:504] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:34.414450 28749 leader_election.cc:288] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:11:34.414902 27760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "132d1b9866574dcea3beed5835e4a22f" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:11:34.414980 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "132d1b9866574dcea3beed5835e4a22f" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:34.415094 27760 raft_consensus.cc:3037] T 132d1b9866574dcea3beed5835e4a22f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:34.415179 28022 raft_consensus.cc:3037] T 132d1b9866574dcea3beed5835e4a22f P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:34.416841 27760 raft_consensus.cc:2454] T 132d1b9866574dcea3beed5835e4a22f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:34.416942 28022 raft_consensus.cc:2454] T 132d1b9866574dcea3beed5835e4a22f P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:34.417238 28087 leader_election.cc:302] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:11:34.417515 28749 raft_consensus.cc:2781] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:34.417681 28749 raft_consensus.cc:686] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:11:34.417979 28749 consensus_queue.cc:237] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:34.420516 27614 catalog_manager.cc:5484] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } }
W20240501 08:11:34.764938 28864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20240501 08:11:34.765210 28749 consensus_queue.cc:1035] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:34.765357 28864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 08:11:34.777451 28805 consensus_queue.cc:1035] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:34.778220 28864 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 08:11:34.836884 28797 consensus_queue.cc:1035] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:34.859269 28805 consensus_queue.cc:1035] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:34.914162 28797 consensus_queue.cc:1035] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:34.921623 28749 consensus_queue.cc:1035] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 308.211 ms
time per row: 0.154106 ms
I20240501 08:11:35.241559 27620 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:35.250638 27872 tablet_service.cc:1467] Processing CreateTablet for tablet 0723fe8697e14b6ea9d2ba92213a8a62 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9a48ae11d98e46f3ab89d4cbc3fe7020]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:35.250793 28131 tablet_service.cc:1467] Processing CreateTablet for tablet 0723fe8697e14b6ea9d2ba92213a8a62 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9a48ae11d98e46f3ab89d4cbc3fe7020]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:35.251132 27872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0723fe8697e14b6ea9d2ba92213a8a62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:35.251174 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0723fe8697e14b6ea9d2ba92213a8a62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:35.251257 27740 tablet_service.cc:1467] Processing CreateTablet for tablet 0723fe8697e14b6ea9d2ba92213a8a62 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9a48ae11d98e46f3ab89d4cbc3fe7020]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:35.251713 27740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0723fe8697e14b6ea9d2ba92213a8a62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:35.253355 27870 tablet_service.cc:1467] Processing CreateTablet for tablet db095644a81a44919887256537e22e70 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9a48ae11d98e46f3ab89d4cbc3fe7020]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:35.253743 27870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db095644a81a44919887256537e22e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:35.253712 27741 tablet_service.cc:1467] Processing CreateTablet for tablet 0c04c587834c408bb8c43a9b57bf1fe6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9a48ae11d98e46f3ab89d4cbc3fe7020]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:35.254118 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c04c587834c408bb8c43a9b57bf1fe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:35.258517 28000 tablet_service.cc:1467] Processing CreateTablet for tablet db095644a81a44919887256537e22e70 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9a48ae11d98e46f3ab89d4cbc3fe7020]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:35.259110 28000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db095644a81a44919887256537e22e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:35.261750 27871 tablet_service.cc:1467] Processing CreateTablet for tablet 0c04c587834c408bb8c43a9b57bf1fe6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9a48ae11d98e46f3ab89d4cbc3fe7020]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:35.262324 27871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c04c587834c408bb8c43a9b57bf1fe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:35.263352 28929 tablet_bootstrap.cc:492] T 0723fe8697e14b6ea9d2ba92213a8a62 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:35.265188 28929 tablet_bootstrap.cc:654] T 0723fe8697e14b6ea9d2ba92213a8a62 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:35.265403 28130 tablet_service.cc:1467] Processing CreateTablet for tablet 0c04c587834c408bb8c43a9b57bf1fe6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9a48ae11d98e46f3ab89d4cbc3fe7020]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:35.265439 28132 tablet_service.cc:1467] Processing CreateTablet for tablet db095644a81a44919887256537e22e70 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9a48ae11d98e46f3ab89d4cbc3fe7020]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:35.265956 28130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c04c587834c408bb8c43a9b57bf1fe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:35.266356 28132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db095644a81a44919887256537e22e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:35.285118 28931 tablet_bootstrap.cc:492] T db095644a81a44919887256537e22e70 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:35.287171 28931 tablet_bootstrap.cc:654] T db095644a81a44919887256537e22e70 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:35.290374 28928 tablet_bootstrap.cc:492] T 0723fe8697e14b6ea9d2ba92213a8a62 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:35.290577 28929 tablet_bootstrap.cc:492] T 0723fe8697e14b6ea9d2ba92213a8a62 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:35.290679 28933 tablet_bootstrap.cc:492] T db095644a81a44919887256537e22e70 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:35.290846 28929 ts_tablet_manager.cc:1367] T 0723fe8697e14b6ea9d2ba92213a8a62 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.022s
I20240501 08:11:35.291217 28931 tablet_bootstrap.cc:492] T db095644a81a44919887256537e22e70 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:35.291371 28931 ts_tablet_manager.cc:1367] T db095644a81a44919887256537e22e70 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 08:11:35.291921 28929 raft_consensus.cc:348] T 0723fe8697e14b6ea9d2ba92213a8a62 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:35.292184 28928 tablet_bootstrap.cc:654] T 0723fe8697e14b6ea9d2ba92213a8a62 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:35.292166 28931 raft_consensus.cc:348] T db095644a81a44919887256537e22e70 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:35.292299 28929 raft_consensus.cc:374] T 0723fe8697e14b6ea9d2ba92213a8a62 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:35.292377 28931 raft_consensus.cc:374] T db095644a81a44919887256537e22e70 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:35.292462 28931 raft_consensus.cc:729] T db095644a81a44919887256537e22e70 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:35.292565 28929 raft_consensus.cc:729] T 0723fe8697e14b6ea9d2ba92213a8a62 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:35.292752 28931 consensus_queue.cc:260] T db095644a81a44919887256537e22e70 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:35.292904 28929 consensus_queue.cc:260] T 0723fe8697e14b6ea9d2ba92213a8a62 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:35.293360 28931 ts_tablet_manager.cc:1398] T db095644a81a44919887256537e22e70 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:35.293689 28931 tablet_bootstrap.cc:492] T 0c04c587834c408bb8c43a9b57bf1fe6 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:35.294231 28929 ts_tablet_manager.cc:1398] T 0723fe8697e14b6ea9d2ba92213a8a62 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:11:35.294636 28929 tablet_bootstrap.cc:492] T 0c04c587834c408bb8c43a9b57bf1fe6 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:35.296877 28929 tablet_bootstrap.cc:654] T 0c04c587834c408bb8c43a9b57bf1fe6 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:35.299780 28931 tablet_bootstrap.cc:654] T 0c04c587834c408bb8c43a9b57bf1fe6 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:35.295861 28933 tablet_bootstrap.cc:654] T db095644a81a44919887256537e22e70 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:35.302121 28928 tablet_bootstrap.cc:492] T 0723fe8697e14b6ea9d2ba92213a8a62 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:35.302345 28928 ts_tablet_manager.cc:1367] T 0723fe8697e14b6ea9d2ba92213a8a62 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.011s
I20240501 08:11:35.302577 28931 tablet_bootstrap.cc:492] T 0c04c587834c408bb8c43a9b57bf1fe6 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:35.302668 28929 tablet_bootstrap.cc:492] T 0c04c587834c408bb8c43a9b57bf1fe6 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:35.302793 28931 ts_tablet_manager.cc:1367] T 0c04c587834c408bb8c43a9b57bf1fe6 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240501 08:11:35.302841 28929 ts_tablet_manager.cc:1367] T 0c04c587834c408bb8c43a9b57bf1fe6 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20240501 08:11:35.303443 28928 raft_consensus.cc:348] T 0723fe8697e14b6ea9d2ba92213a8a62 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:35.303534 28929 raft_consensus.cc:348] T 0c04c587834c408bb8c43a9b57bf1fe6 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:35.303754 28929 raft_consensus.cc:374] T 0c04c587834c408bb8c43a9b57bf1fe6 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:35.303768 28928 raft_consensus.cc:374] T 0723fe8697e14b6ea9d2ba92213a8a62 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:35.303843 28929 raft_consensus.cc:729] T 0c04c587834c408bb8c43a9b57bf1fe6 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:35.303803 28931 raft_consensus.cc:348] T 0c04c587834c408bb8c43a9b57bf1fe6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:35.303987 28928 raft_consensus.cc:729] T 0723fe8697e14b6ea9d2ba92213a8a62 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:35.304090 28931 raft_consensus.cc:374] T 0c04c587834c408bb8c43a9b57bf1fe6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:35.304189 28931 raft_consensus.cc:729] T 0c04c587834c408bb8c43a9b57bf1fe6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:35.304119 28929 consensus_queue.cc:260] T 0c04c587834c408bb8c43a9b57bf1fe6 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:35.304488 28931 consensus_queue.cc:260] T 0c04c587834c408bb8c43a9b57bf1fe6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:35.304692 28929 ts_tablet_manager.cc:1398] T 0c04c587834c408bb8c43a9b57bf1fe6 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:11:35.305022 28931 ts_tablet_manager.cc:1398] T 0c04c587834c408bb8c43a9b57bf1fe6 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:11:35.304976 28928 consensus_queue.cc:260] T 0723fe8697e14b6ea9d2ba92213a8a62 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:35.305567 28931 tablet_bootstrap.cc:492] T 0723fe8697e14b6ea9d2ba92213a8a62 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:35.306272 28928 ts_tablet_manager.cc:1398] T 0723fe8697e14b6ea9d2ba92213a8a62 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:11:35.306797 28928 tablet_bootstrap.cc:492] T db095644a81a44919887256537e22e70 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:35.309010 28931 tablet_bootstrap.cc:654] T 0723fe8697e14b6ea9d2ba92213a8a62 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:35.309733 28928 tablet_bootstrap.cc:654] T db095644a81a44919887256537e22e70 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:35.311801 28928 tablet_bootstrap.cc:492] T db095644a81a44919887256537e22e70 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:35.311986 28928 ts_tablet_manager.cc:1367] T db095644a81a44919887256537e22e70 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240501 08:11:35.312568 28928 raft_consensus.cc:348] T db095644a81a44919887256537e22e70 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:35.312764 28928 raft_consensus.cc:374] T db095644a81a44919887256537e22e70 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:35.312846 28928 raft_consensus.cc:729] T db095644a81a44919887256537e22e70 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:35.313097 28928 consensus_queue.cc:260] T db095644a81a44919887256537e22e70 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:35.313243 28933 tablet_bootstrap.cc:492] T db095644a81a44919887256537e22e70 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:35.313442 28933 ts_tablet_manager.cc:1367] T db095644a81a44919887256537e22e70 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.003s
I20240501 08:11:35.313701 28928 ts_tablet_manager.cc:1398] T db095644a81a44919887256537e22e70 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:11:35.314527 28928 tablet_bootstrap.cc:492] T 0c04c587834c408bb8c43a9b57bf1fe6 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:35.315464 28931 tablet_bootstrap.cc:492] T 0723fe8697e14b6ea9d2ba92213a8a62 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:35.315630 28931 ts_tablet_manager.cc:1367] T 0723fe8697e14b6ea9d2ba92213a8a62 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20240501 08:11:35.316160 28928 tablet_bootstrap.cc:654] T 0c04c587834c408bb8c43a9b57bf1fe6 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:35.316058 28933 raft_consensus.cc:348] T db095644a81a44919887256537e22e70 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:35.316274 28933 raft_consensus.cc:374] T db095644a81a44919887256537e22e70 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:35.316360 28933 raft_consensus.cc:729] T db095644a81a44919887256537e22e70 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:35.316409 28931 raft_consensus.cc:348] T 0723fe8697e14b6ea9d2ba92213a8a62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:35.316583 28931 raft_consensus.cc:374] T 0723fe8697e14b6ea9d2ba92213a8a62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:35.316675 28931 raft_consensus.cc:729] T 0723fe8697e14b6ea9d2ba92213a8a62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:35.316642 28933 consensus_queue.cc:260] T db095644a81a44919887256537e22e70 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:35.316923 28931 consensus_queue.cc:260] T 0723fe8697e14b6ea9d2ba92213a8a62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:35.317507 28931 ts_tablet_manager.cc:1398] T 0723fe8697e14b6ea9d2ba92213a8a62 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:11:35.318383 28933 ts_tablet_manager.cc:1398] T db095644a81a44919887256537e22e70 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240501 08:11:35.319314 28928 tablet_bootstrap.cc:492] T 0c04c587834c408bb8c43a9b57bf1fe6 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:35.319455 28928 ts_tablet_manager.cc:1367] T 0c04c587834c408bb8c43a9b57bf1fe6 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240501 08:11:35.320188 28928 raft_consensus.cc:348] T 0c04c587834c408bb8c43a9b57bf1fe6 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:35.320395 28928 raft_consensus.cc:374] T 0c04c587834c408bb8c43a9b57bf1fe6 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:35.320494 28928 raft_consensus.cc:729] T 0c04c587834c408bb8c43a9b57bf1fe6 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:35.320748 28928 consensus_queue.cc:260] T 0c04c587834c408bb8c43a9b57bf1fe6 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:35.321816 28928 ts_tablet_manager.cc:1398] T 0c04c587834c408bb8c43a9b57bf1fe6 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:11:35.395526 28797 raft_consensus.cc:482] T db095644a81a44919887256537e22e70 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:35.395776 28797 raft_consensus.cc:504] T db095644a81a44919887256537e22e70 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:35.396420 28797 leader_election.cc:288] T db095644a81a44919887256537e22e70 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:11:35.397056 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db095644a81a44919887256537e22e70" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:35.397318 28021 raft_consensus.cc:2454] T db095644a81a44919887256537e22e70 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:11:35.397763 27827 leader_election.cc:302] T db095644a81a44919887256537e22e70 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aad81047009a4ffcbc450b60030583b5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:35.398154 28797 raft_consensus.cc:2781] T db095644a81a44919887256537e22e70 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:35.398248 28797 raft_consensus.cc:482] T db095644a81a44919887256537e22e70 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:35.398355 28797 raft_consensus.cc:3037] T db095644a81a44919887256537e22e70 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:35.400785 28797 raft_consensus.cc:504] T db095644a81a44919887256537e22e70 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:35.401499 28797 leader_election.cc:288] T db095644a81a44919887256537e22e70 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Requested vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:11:35.401892 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db095644a81a44919887256537e22e70" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:35.402192 28021 raft_consensus.cc:3037] T db095644a81a44919887256537e22e70 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:35.403681 28152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db095644a81a44919887256537e22e70" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:11:35.403932 28152 raft_consensus.cc:2454] T db095644a81a44919887256537e22e70 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:11:35.403856 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db095644a81a44919887256537e22e70" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:11:35.404157 28151 raft_consensus.cc:3037] T db095644a81a44919887256537e22e70 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:35.404407 28021 raft_consensus.cc:2454] T db095644a81a44919887256537e22e70 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:11:35.404803 27827 leader_election.cc:302] T db095644a81a44919887256537e22e70 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aad81047009a4ffcbc450b60030583b5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:35.405076 28797 raft_consensus.cc:2781] T db095644a81a44919887256537e22e70 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:35.405222 28797 raft_consensus.cc:686] T db095644a81a44919887256537e22e70 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Becoming Leader. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Running, Role: LEADER
I20240501 08:11:35.405457 28797 consensus_queue.cc:237] T db095644a81a44919887256537e22e70 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:35.406630 28151 raft_consensus.cc:2454] T db095644a81a44919887256537e22e70 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:11:35.408206 27615 catalog_manager.cc:5484] T db095644a81a44919887256537e22e70 P f1525cbc82a24d13b65f136695c903af reported cstate change: term changed from 0 to 1, leader changed from <none> to f1525cbc82a24d13b65f136695c903af (127.21.84.130). New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:35.455322 28805 raft_consensus.cc:482] T 0c04c587834c408bb8c43a9b57bf1fe6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:35.455516 28805 raft_consensus.cc:504] T 0c04c587834c408bb8c43a9b57bf1fe6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:35.456233 28805 leader_election.cc:288] T 0c04c587834c408bb8c43a9b57bf1fe6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:11:35.456820 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c04c587834c408bb8c43a9b57bf1fe6" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:11:35.456848 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c04c587834c408bb8c43a9b57bf1fe6" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:11:35.457044 27761 raft_consensus.cc:2454] T 0c04c587834c408bb8c43a9b57bf1fe6 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:35.457067 27890 raft_consensus.cc:2454] T 0c04c587834c408bb8c43a9b57bf1fe6 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:35.457497 28088 leader_election.cc:302] T 0c04c587834c408bb8c43a9b57bf1fe6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:35.457780 28805 raft_consensus.cc:2781] T 0c04c587834c408bb8c43a9b57bf1fe6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:35.457880 28805 raft_consensus.cc:482] T 0c04c587834c408bb8c43a9b57bf1fe6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:35.457983 28805 raft_consensus.cc:3037] T 0c04c587834c408bb8c43a9b57bf1fe6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:35.459884 28805 raft_consensus.cc:504] T 0c04c587834c408bb8c43a9b57bf1fe6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:35.460490 28805 leader_election.cc:288] T 0c04c587834c408bb8c43a9b57bf1fe6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:11:35.461021 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c04c587834c408bb8c43a9b57bf1fe6" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:11:35.461068 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c04c587834c408bb8c43a9b57bf1fe6" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:11:35.461246 27761 raft_consensus.cc:3037] T 0c04c587834c408bb8c43a9b57bf1fe6 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:35.461283 27890 raft_consensus.cc:3037] T 0c04c587834c408bb8c43a9b57bf1fe6 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:35.463238 27761 raft_consensus.cc:2454] T 0c04c587834c408bb8c43a9b57bf1fe6 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:35.463328 27890 raft_consensus.cc:2454] T 0c04c587834c408bb8c43a9b57bf1fe6 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:35.463707 28087 leader_election.cc:302] T 0c04c587834c408bb8c43a9b57bf1fe6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:11:35.464044 28805 raft_consensus.cc:2781] T 0c04c587834c408bb8c43a9b57bf1fe6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:35.464177 28805 raft_consensus.cc:686] T 0c04c587834c408bb8c43a9b57bf1fe6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:11:35.464478 28805 consensus_queue.cc:237] T 0c04c587834c408bb8c43a9b57bf1fe6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:35.467103 27615 catalog_manager.cc:5484] T 0c04c587834c408bb8c43a9b57bf1fe6 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:35.546989 28764 raft_consensus.cc:482] T 0723fe8697e14b6ea9d2ba92213a8a62 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:35.547176 28764 raft_consensus.cc:504] T 0723fe8697e14b6ea9d2ba92213a8a62 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:35.547915 28764 leader_election.cc:288] T 0723fe8697e14b6ea9d2ba92213a8a62 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:11:35.548429 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0723fe8697e14b6ea9d2ba92213a8a62" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:11:35.548460 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0723fe8697e14b6ea9d2ba92213a8a62" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:11:35.548712 27890 raft_consensus.cc:2454] T 0723fe8697e14b6ea9d2ba92213a8a62 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:11:35.548715 28151 raft_consensus.cc:2454] T 0723fe8697e14b6ea9d2ba92213a8a62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:11:35.549157 27694 leader_election.cc:302] T 0723fe8697e14b6ea9d2ba92213a8a62 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:11:35.549415 28764 raft_consensus.cc:2781] T 0723fe8697e14b6ea9d2ba92213a8a62 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:35.549541 28764 raft_consensus.cc:482] T 0723fe8697e14b6ea9d2ba92213a8a62 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:35.549630 28764 raft_consensus.cc:3037] T 0723fe8697e14b6ea9d2ba92213a8a62 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:35.551533 28764 raft_consensus.cc:504] T 0723fe8697e14b6ea9d2ba92213a8a62 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:35.552110 28764 leader_election.cc:288] T 0723fe8697e14b6ea9d2ba92213a8a62 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Requested vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:11:35.552656 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0723fe8697e14b6ea9d2ba92213a8a62" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:11:35.552654 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0723fe8697e14b6ea9d2ba92213a8a62" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:11:35.552901 28151 raft_consensus.cc:3037] T 0723fe8697e14b6ea9d2ba92213a8a62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:35.552901 27890 raft_consensus.cc:3037] T 0723fe8697e14b6ea9d2ba92213a8a62 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:35.555796 28151 raft_consensus.cc:2454] T 0723fe8697e14b6ea9d2ba92213a8a62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:11:35.555796 27890 raft_consensus.cc:2454] T 0723fe8697e14b6ea9d2ba92213a8a62 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:11:35.556399 27697 leader_election.cc:302] T 0723fe8697e14b6ea9d2ba92213a8a62 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:35.556797 28764 raft_consensus.cc:2781] T 0723fe8697e14b6ea9d2ba92213a8a62 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:35.557241 28764 raft_consensus.cc:686] T 0723fe8697e14b6ea9d2ba92213a8a62 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:11:35.557610 28764 consensus_queue.cc:237] T 0723fe8697e14b6ea9d2ba92213a8a62 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:35.560511 27615 catalog_manager.cc:5484] T 0723fe8697e14b6ea9d2ba92213a8a62 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:35.789466 28797 consensus_queue.cc:1035] T db095644a81a44919887256537e22e70 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:35.794927 28797 consensus_queue.cc:1035] T db095644a81a44919887256537e22e70 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:35.919715 28946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:35.920030 28946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:35.927301 28946 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 08:11:35.973304 28805 consensus_queue.cc:1035] T 0c04c587834c408bb8c43a9b57bf1fe6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:35.978555 28805 consensus_queue.cc:1035] T 0c04c587834c408bb8c43a9b57bf1fe6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:36.018069 28764 consensus_queue.cc:1035] T 0723fe8697e14b6ea9d2ba92213a8a62 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:36.034756 28764 consensus_queue.cc:1035] T 0723fe8697e14b6ea9d2ba92213a8a62 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 254.376 ms
time per row: 0.127188 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 08:11:37.244825 27613 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:37.254400 28000 tablet_service.cc:1467] Processing CreateTablet for tablet ac4914417d9e44e7a9591be5386f1d8b (DEFAULT_TABLE table=rebalance_test_table_3 [id=56fa27c3d3de4793aadbcaf54cdb0ac5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:37.254385 27741 tablet_service.cc:1467] Processing CreateTablet for tablet ac4914417d9e44e7a9591be5386f1d8b (DEFAULT_TABLE table=rebalance_test_table_3 [id=56fa27c3d3de4793aadbcaf54cdb0ac5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:37.254856 28000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac4914417d9e44e7a9591be5386f1d8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:37.254856 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac4914417d9e44e7a9591be5386f1d8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:37.259056 27872 tablet_service.cc:1467] Processing CreateTablet for tablet ac4914417d9e44e7a9591be5386f1d8b (DEFAULT_TABLE table=rebalance_test_table_3 [id=56fa27c3d3de4793aadbcaf54cdb0ac5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:37.259506 27872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac4914417d9e44e7a9591be5386f1d8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:37.262210 28131 tablet_service.cc:1467] Processing CreateTablet for tablet f2baff8656804ce2bfb64a9280e16855 (DEFAULT_TABLE table=rebalance_test_table_3 [id=56fa27c3d3de4793aadbcaf54cdb0ac5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:37.262666 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2baff8656804ce2bfb64a9280e16855. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:37.262789 27740 tablet_service.cc:1467] Processing CreateTablet for tablet f2baff8656804ce2bfb64a9280e16855 (DEFAULT_TABLE table=rebalance_test_table_3 [id=56fa27c3d3de4793aadbcaf54cdb0ac5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:37.263315 27740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2baff8656804ce2bfb64a9280e16855. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:37.266517 28001 tablet_service.cc:1467] Processing CreateTablet for tablet f2baff8656804ce2bfb64a9280e16855 (DEFAULT_TABLE table=rebalance_test_table_3 [id=56fa27c3d3de4793aadbcaf54cdb0ac5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:37.267109 28001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2baff8656804ce2bfb64a9280e16855. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:37.267591 28002 tablet_service.cc:1467] Processing CreateTablet for tablet 4b3e3d39f82549079293d98357884bf0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=56fa27c3d3de4793aadbcaf54cdb0ac5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:37.267969 28002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b3e3d39f82549079293d98357884bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:37.271448 28130 tablet_service.cc:1467] Processing CreateTablet for tablet 4b3e3d39f82549079293d98357884bf0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=56fa27c3d3de4793aadbcaf54cdb0ac5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:37.272027 28130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b3e3d39f82549079293d98357884bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:37.273825 29009 tablet_bootstrap.cc:492] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:37.274389 27871 tablet_service.cc:1467] Processing CreateTablet for tablet 4b3e3d39f82549079293d98357884bf0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=56fa27c3d3de4793aadbcaf54cdb0ac5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:37.274822 27871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b3e3d39f82549079293d98357884bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:37.275805 29009 tablet_bootstrap.cc:654] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:37.283871 29011 tablet_bootstrap.cc:492] T f2baff8656804ce2bfb64a9280e16855 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:37.285775 29011 tablet_bootstrap.cc:654] T f2baff8656804ce2bfb64a9280e16855 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:37.287560 29010 tablet_bootstrap.cc:492] T ac4914417d9e44e7a9591be5386f1d8b P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:37.289068 29010 tablet_bootstrap.cc:654] T ac4914417d9e44e7a9591be5386f1d8b P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:37.296015 29010 tablet_bootstrap.cc:492] T ac4914417d9e44e7a9591be5386f1d8b P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:37.296345 29010 ts_tablet_manager.cc:1367] T ac4914417d9e44e7a9591be5386f1d8b P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20240501 08:11:37.297495 29010 raft_consensus.cc:348] T ac4914417d9e44e7a9591be5386f1d8b P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:37.297734 29015 tablet_bootstrap.cc:492] T f2baff8656804ce2bfb64a9280e16855 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:37.297996 29010 raft_consensus.cc:374] T ac4914417d9e44e7a9591be5386f1d8b P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:37.298146 29010 raft_consensus.cc:729] T ac4914417d9e44e7a9591be5386f1d8b P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:37.298517 29010 consensus_queue.cc:260] T ac4914417d9e44e7a9591be5386f1d8b P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:37.299319 29010 ts_tablet_manager.cc:1398] T ac4914417d9e44e7a9591be5386f1d8b P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:37.299712 29010 tablet_bootstrap.cc:492] T 4b3e3d39f82549079293d98357884bf0 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:37.302210 29010 tablet_bootstrap.cc:654] T 4b3e3d39f82549079293d98357884bf0 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:37.303966 29011 tablet_bootstrap.cc:492] T f2baff8656804ce2bfb64a9280e16855 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:37.304255 29009 tablet_bootstrap.cc:492] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:37.304244 29011 ts_tablet_manager.cc:1367] T f2baff8656804ce2bfb64a9280e16855 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.003s
I20240501 08:11:37.304421 29009 ts_tablet_manager.cc:1367] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.031s user 0.018s sys 0.005s
I20240501 08:11:37.305248 29015 tablet_bootstrap.cc:654] T f2baff8656804ce2bfb64a9280e16855 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:37.305182 29009 raft_consensus.cc:348] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:37.305442 29009 raft_consensus.cc:374] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:37.305578 29009 raft_consensus.cc:729] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:37.305392 29011 raft_consensus.cc:348] T f2baff8656804ce2bfb64a9280e16855 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:37.305697 29011 raft_consensus.cc:374] T f2baff8656804ce2bfb64a9280e16855 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:37.305857 29011 raft_consensus.cc:729] T f2baff8656804ce2bfb64a9280e16855 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:37.305920 29009 consensus_queue.cc:260] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:37.306238 29011 consensus_queue.cc:260] T f2baff8656804ce2bfb64a9280e16855 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:37.306543 29009 ts_tablet_manager.cc:1398] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:37.306818 29011 ts_tablet_manager.cc:1398] T f2baff8656804ce2bfb64a9280e16855 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:11:37.306962 29009 tablet_bootstrap.cc:492] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:37.308646 29009 tablet_bootstrap.cc:654] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:37.312096 29010 tablet_bootstrap.cc:492] T 4b3e3d39f82549079293d98357884bf0 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:37.312461 29011 tablet_bootstrap.cc:492] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:37.312516 29010 ts_tablet_manager.cc:1367] T 4b3e3d39f82549079293d98357884bf0 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20240501 08:11:37.313366 29010 raft_consensus.cc:348] T 4b3e3d39f82549079293d98357884bf0 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:37.313705 29010 raft_consensus.cc:374] T 4b3e3d39f82549079293d98357884bf0 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:37.313832 29010 raft_consensus.cc:729] T 4b3e3d39f82549079293d98357884bf0 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:37.314239 29011 tablet_bootstrap.cc:654] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:37.314193 29010 consensus_queue.cc:260] T 4b3e3d39f82549079293d98357884bf0 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:37.315053 29010 ts_tablet_manager.cc:1398] T 4b3e3d39f82549079293d98357884bf0 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:37.317302 29015 tablet_bootstrap.cc:492] T f2baff8656804ce2bfb64a9280e16855 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:37.317582 29015 ts_tablet_manager.cc:1367] T f2baff8656804ce2bfb64a9280e16855 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.005s
I20240501 08:11:37.319527 29009 tablet_bootstrap.cc:492] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:37.319422 29015 raft_consensus.cc:348] T f2baff8656804ce2bfb64a9280e16855 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:37.319741 29015 raft_consensus.cc:374] T f2baff8656804ce2bfb64a9280e16855 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:37.319757 29009 ts_tablet_manager.cc:1367] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20240501 08:11:37.319911 29015 raft_consensus.cc:729] T f2baff8656804ce2bfb64a9280e16855 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:37.320226 29015 consensus_queue.cc:260] T f2baff8656804ce2bfb64a9280e16855 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:37.320523 29009 raft_consensus.cc:348] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:37.320850 29009 raft_consensus.cc:374] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:37.320976 29009 raft_consensus.cc:729] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:37.321089 29011 tablet_bootstrap.cc:492] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:37.321324 29011 ts_tablet_manager.cc:1367] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240501 08:11:37.321332 29009 consensus_queue.cc:260] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:37.322110 29015 ts_tablet_manager.cc:1398] T f2baff8656804ce2bfb64a9280e16855 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20240501 08:11:37.322417 29009 ts_tablet_manager.cc:1398] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:11:37.322543 29015 tablet_bootstrap.cc:492] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:37.323352 29011 raft_consensus.cc:348] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:37.323596 29011 raft_consensus.cc:374] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:37.323683 29011 raft_consensus.cc:729] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:37.323957 29011 consensus_queue.cc:260] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:37.324326 29015 tablet_bootstrap.cc:654] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:37.324537 29011 ts_tablet_manager.cc:1398] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:11:37.327862 29015 tablet_bootstrap.cc:492] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:37.328050 29015 ts_tablet_manager.cc:1367] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240501 08:11:37.328886 29015 raft_consensus.cc:348] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:37.329157 29015 raft_consensus.cc:374] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:37.329252 29015 raft_consensus.cc:729] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:37.329490 29015 consensus_queue.cc:260] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:37.330027 29015 ts_tablet_manager.cc:1398] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:37.330327 29015 tablet_bootstrap.cc:492] T ac4914417d9e44e7a9591be5386f1d8b P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:37.331753 29015 tablet_bootstrap.cc:654] T ac4914417d9e44e7a9591be5386f1d8b P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:37.333941 29015 tablet_bootstrap.cc:492] T ac4914417d9e44e7a9591be5386f1d8b P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:37.334117 29015 ts_tablet_manager.cc:1367] T ac4914417d9e44e7a9591be5386f1d8b P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240501 08:11:37.334864 29015 raft_consensus.cc:348] T ac4914417d9e44e7a9591be5386f1d8b P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:37.335027 29015 raft_consensus.cc:374] T ac4914417d9e44e7a9591be5386f1d8b P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:37.335091 29015 raft_consensus.cc:729] T ac4914417d9e44e7a9591be5386f1d8b P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:37.335283 29015 consensus_queue.cc:260] T ac4914417d9e44e7a9591be5386f1d8b P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:37.335707 29015 ts_tablet_manager.cc:1398] T ac4914417d9e44e7a9591be5386f1d8b P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:11:37.437773 28944 raft_consensus.cc:482] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:37.438009 28944 raft_consensus.cc:504] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:37.438659 28944 leader_election.cc:288] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:11:37.439209 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4914417d9e44e7a9591be5386f1d8b" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:11:37.439213 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4914417d9e44e7a9591be5386f1d8b" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:37.439436 27890 raft_consensus.cc:2454] T ac4914417d9e44e7a9591be5386f1d8b P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:11:37.439451 28021 raft_consensus.cc:2454] T ac4914417d9e44e7a9591be5386f1d8b P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:11:37.439911 27696 leader_election.cc:302] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:37.440171 28944 raft_consensus.cc:2781] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:37.440289 28944 raft_consensus.cc:482] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:37.440362 28944 raft_consensus.cc:3037] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:37.442243 28944 raft_consensus.cc:504] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:37.442807 28944 leader_election.cc:288] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Requested vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:11:37.443315 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4914417d9e44e7a9591be5386f1d8b" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:37.443344 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4914417d9e44e7a9591be5386f1d8b" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:11:37.443503 28021 raft_consensus.cc:3037] T ac4914417d9e44e7a9591be5386f1d8b P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:37.443506 27890 raft_consensus.cc:3037] T ac4914417d9e44e7a9591be5386f1d8b P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:37.445290 27890 raft_consensus.cc:2454] T ac4914417d9e44e7a9591be5386f1d8b P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:11:37.445304 28021 raft_consensus.cc:2454] T ac4914417d9e44e7a9591be5386f1d8b P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:11:37.445708 27696 leader_election.cc:302] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:37.446024 28944 raft_consensus.cc:2781] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:37.446332 28944 raft_consensus.cc:686] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:11:37.446614 28944 consensus_queue.cc:237] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:37.449311 27617 catalog_manager.cc:5484] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:37.497678 28944 raft_consensus.cc:482] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:37.497973 28944 raft_consensus.cc:504] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:37.498615 28944 leader_election.cc:288] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:11:37.499212 28152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2baff8656804ce2bfb64a9280e16855" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:11:37.499280 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2baff8656804ce2bfb64a9280e16855" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:37.499476 28152 raft_consensus.cc:2454] T f2baff8656804ce2bfb64a9280e16855 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:11:37.499572 28021 raft_consensus.cc:2454] T f2baff8656804ce2bfb64a9280e16855 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:11:37.500064 27696 leader_election.cc:302] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:37.500391 28944 raft_consensus.cc:2781] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:37.500490 28944 raft_consensus.cc:482] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:37.500594 28944 raft_consensus.cc:3037] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:37.502486 28944 raft_consensus.cc:504] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:37.503018 28944 leader_election.cc:288] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Requested vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:11:37.503566 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2baff8656804ce2bfb64a9280e16855" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:37.503566 28152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2baff8656804ce2bfb64a9280e16855" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:11:37.503803 28152 raft_consensus.cc:3037] T f2baff8656804ce2bfb64a9280e16855 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:37.503804 28021 raft_consensus.cc:3037] T f2baff8656804ce2bfb64a9280e16855 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:37.506800 28152 raft_consensus.cc:2454] T f2baff8656804ce2bfb64a9280e16855 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:11:37.506781 28021 raft_consensus.cc:2454] T f2baff8656804ce2bfb64a9280e16855 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:11:37.507217 27696 leader_election.cc:302] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:37.507544 28944 raft_consensus.cc:2781] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:37.507727 28944 raft_consensus.cc:686] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:11:37.508100 28944 consensus_queue.cc:237] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:37.511116 27617 catalog_manager.cc:5484] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:37.719079 29020 raft_consensus.cc:482] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:37.719318 29020 raft_consensus.cc:504] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:37.720973 29020 leader_election.cc:288] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:11:37.737455 28992 raft_consensus.cc:482] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:37.737684 28992 raft_consensus.cc:504] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:37.738466 28992 leader_election.cc:288] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:11:37.739490 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b3e3d39f82549079293d98357884bf0" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:11:37.739768 27890 raft_consensus.cc:2454] T 4b3e3d39f82549079293d98357884bf0 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:37.740247 28088 leader_election.cc:302] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:37.741312 29030 raft_consensus.cc:2781] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:37.741485 29030 raft_consensus.cc:482] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:37.741681 29030 raft_consensus.cc:3037] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:37.741797 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b3e3d39f82549079293d98357884bf0" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:37.742117 28021 raft_consensus.cc:2454] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:37.744596 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b3e3d39f82549079293d98357884bf0" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:11:37.744853 27890 raft_consensus.cc:2454] T 4b3e3d39f82549079293d98357884bf0 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 0.
I20240501 08:11:37.744767 29030 raft_consensus.cc:504] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:37.745323 27958 leader_election.cc:302] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aad81047009a4ffcbc450b60030583b5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:37.745810 29020 raft_consensus.cc:2781] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:37.745972 29020 raft_consensus.cc:482] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:37.745890 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b3e3d39f82549079293d98357884bf0" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:37.746098 29020 raft_consensus.cc:3037] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:37.746485 29030 leader_election.cc:288] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:11:37.746737 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b3e3d39f82549079293d98357884bf0" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:11:37.746940 27890 raft_consensus.cc:3037] T 4b3e3d39f82549079293d98357884bf0 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:37.748996 29020 raft_consensus.cc:504] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:37.749480 28021 raft_consensus.cc:2379] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in current term 1: Already voted for candidate aad81047009a4ffcbc450b60030583b5 in this term.
I20240501 08:11:37.749637 27890 raft_consensus.cc:2454] T 4b3e3d39f82549079293d98357884bf0 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:37.749861 28152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b3e3d39f82549079293d98357884bf0" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:11:37.750025 29020 leader_election.cc:288] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Requested vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:11:37.750072 28088 leader_election.cc:302] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:37.750211 28152 raft_consensus.cc:2379] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate aad81047009a4ffcbc450b60030583b5 in current term 1: Already voted for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in this term.
I20240501 08:11:37.750497 29030 raft_consensus.cc:2781] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:37.750725 29030 raft_consensus.cc:686] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:11:37.750661 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b3e3d39f82549079293d98357884bf0" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:11:37.750921 27890 raft_consensus.cc:2379] T 4b3e3d39f82549079293d98357884bf0 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate aad81047009a4ffcbc450b60030583b5 in current term 1: Already voted for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in this term.
I20240501 08:11:37.751061 28152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b3e3d39f82549079293d98357884bf0" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:11:37.751430 29030 consensus_queue.cc:237] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:37.751523 27958 leader_election.cc:302] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aad81047009a4ffcbc450b60030583b5; no voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af
I20240501 08:11:37.751850 29020 raft_consensus.cc:2726] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 08:11:37.754616 27617 catalog_manager.cc:5484] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:37.923401 28944 consensus_queue.cc:1035] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:37.929695 29023 consensus_queue.cc:1035] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:38.009869 29023 consensus_queue.cc:1035] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:38.015465 29037 consensus_queue.cc:1035] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:38.173434 29030 consensus_queue.cc:1035] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:38.178756 28992 consensus_queue.cc:1035] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:38.180168 29033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:38.180625 29033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:38.189046 29033 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 226.613 ms
time per row: 0.113307 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 08:11:39.489672 27613 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:39.498402 28130 tablet_service.cc:1467] Processing CreateTablet for tablet 60fb22f7cdeb4c979377537877845c16 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9c7af050a91047f384bcfb6909198411]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:39.498494 27871 tablet_service.cc:1467] Processing CreateTablet for tablet 60fb22f7cdeb4c979377537877845c16 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9c7af050a91047f384bcfb6909198411]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:39.498917 28130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60fb22f7cdeb4c979377537877845c16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:39.498920 27871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60fb22f7cdeb4c979377537877845c16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:39.499027 28000 tablet_service.cc:1467] Processing CreateTablet for tablet 24d7b7f928ad446d8fe0d7cdc507c620 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9c7af050a91047f384bcfb6909198411]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:39.499445 28000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24d7b7f928ad446d8fe0d7cdc507c620. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:39.503983 29104 tablet_bootstrap.cc:492] T 24d7b7f928ad446d8fe0d7cdc507c620 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:39.504561 28001 tablet_service.cc:1467] Processing CreateTablet for tablet 251e073eb0e44b02bb4ef9cb98aded4b (DEFAULT_TABLE table=rebalance_test_table_4 [id=9c7af050a91047f384bcfb6909198411]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:39.504978 28001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 251e073eb0e44b02bb4ef9cb98aded4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:39.505740 29104 tablet_bootstrap.cc:654] T 24d7b7f928ad446d8fe0d7cdc507c620 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:39.506078 27741 tablet_service.cc:1467] Processing CreateTablet for tablet 24d7b7f928ad446d8fe0d7cdc507c620 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9c7af050a91047f384bcfb6909198411]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:39.506592 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24d7b7f928ad446d8fe0d7cdc507c620. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:39.510489 27740 tablet_service.cc:1467] Processing CreateTablet for tablet 60fb22f7cdeb4c979377537877845c16 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9c7af050a91047f384bcfb6909198411]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:39.511022 27740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60fb22f7cdeb4c979377537877845c16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:39.510481 27739 tablet_service.cc:1467] Processing CreateTablet for tablet 251e073eb0e44b02bb4ef9cb98aded4b (DEFAULT_TABLE table=rebalance_test_table_4 [id=9c7af050a91047f384bcfb6909198411]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:39.511546 27739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 251e073eb0e44b02bb4ef9cb98aded4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:39.511510 27872 tablet_service.cc:1467] Processing CreateTablet for tablet 24d7b7f928ad446d8fe0d7cdc507c620 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9c7af050a91047f384bcfb6909198411]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:39.511888 27872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24d7b7f928ad446d8fe0d7cdc507c620. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:39.515841 28131 tablet_service.cc:1467] Processing CreateTablet for tablet 251e073eb0e44b02bb4ef9cb98aded4b (DEFAULT_TABLE table=rebalance_test_table_4 [id=9c7af050a91047f384bcfb6909198411]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:39.516247 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 251e073eb0e44b02bb4ef9cb98aded4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:39.528031 29106 tablet_bootstrap.cc:492] T 60fb22f7cdeb4c979377537877845c16 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:39.528141 29104 tablet_bootstrap.cc:492] T 24d7b7f928ad446d8fe0d7cdc507c620 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:39.528280 29104 ts_tablet_manager.cc:1367] T 24d7b7f928ad446d8fe0d7cdc507c620 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.012s
I20240501 08:11:39.529035 29104 raft_consensus.cc:348] T 24d7b7f928ad446d8fe0d7cdc507c620 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:39.529290 29104 raft_consensus.cc:374] T 24d7b7f928ad446d8fe0d7cdc507c620 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:39.529394 29104 raft_consensus.cc:729] T 24d7b7f928ad446d8fe0d7cdc507c620 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:39.530056 29104 consensus_queue.cc:260] T 24d7b7f928ad446d8fe0d7cdc507c620 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:39.530426 29106 tablet_bootstrap.cc:654] T 60fb22f7cdeb4c979377537877845c16 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:39.531009 29108 tablet_bootstrap.cc:492] T 251e073eb0e44b02bb4ef9cb98aded4b P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:39.532156 29105 tablet_bootstrap.cc:492] T 60fb22f7cdeb4c979377537877845c16 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:39.532779 29104 ts_tablet_manager.cc:1398] T 24d7b7f928ad446d8fe0d7cdc507c620 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 08:11:39.533226 29104 tablet_bootstrap.cc:492] T 251e073eb0e44b02bb4ef9cb98aded4b P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:39.533232 29108 tablet_bootstrap.cc:654] T 251e073eb0e44b02bb4ef9cb98aded4b P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:39.534382 29105 tablet_bootstrap.cc:654] T 60fb22f7cdeb4c979377537877845c16 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:39.534868 29104 tablet_bootstrap.cc:654] T 251e073eb0e44b02bb4ef9cb98aded4b P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:39.535379 29106 tablet_bootstrap.cc:492] T 60fb22f7cdeb4c979377537877845c16 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:39.535529 29106 ts_tablet_manager.cc:1367] T 60fb22f7cdeb4c979377537877845c16 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240501 08:11:39.536597 29106 raft_consensus.cc:348] T 60fb22f7cdeb4c979377537877845c16 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:39.536960 29106 raft_consensus.cc:374] T 60fb22f7cdeb4c979377537877845c16 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:39.537117 29106 raft_consensus.cc:729] T 60fb22f7cdeb4c979377537877845c16 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:39.537498 29106 consensus_queue.cc:260] T 60fb22f7cdeb4c979377537877845c16 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:39.538470 29106 ts_tablet_manager.cc:1398] T 60fb22f7cdeb4c979377537877845c16 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 08:11:39.538902 29106 tablet_bootstrap.cc:492] T 24d7b7f928ad446d8fe0d7cdc507c620 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:39.543272 29106 tablet_bootstrap.cc:654] T 24d7b7f928ad446d8fe0d7cdc507c620 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:39.551326 29106 tablet_bootstrap.cc:492] T 24d7b7f928ad446d8fe0d7cdc507c620 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:39.551651 29106 ts_tablet_manager.cc:1367] T 24d7b7f928ad446d8fe0d7cdc507c620 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.004s
I20240501 08:11:39.553005 29106 raft_consensus.cc:348] T 24d7b7f928ad446d8fe0d7cdc507c620 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:39.554700 29105 tablet_bootstrap.cc:492] T 60fb22f7cdeb4c979377537877845c16 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:39.555011 29105 ts_tablet_manager.cc:1367] T 60fb22f7cdeb4c979377537877845c16 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.000s
I20240501 08:11:39.556108 29105 raft_consensus.cc:348] T 60fb22f7cdeb4c979377537877845c16 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:39.556524 29105 raft_consensus.cc:374] T 60fb22f7cdeb4c979377537877845c16 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:39.556687 29105 raft_consensus.cc:729] T 60fb22f7cdeb4c979377537877845c16 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:39.557106 29105 consensus_queue.cc:260] T 60fb22f7cdeb4c979377537877845c16 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:39.558090 29105 ts_tablet_manager.cc:1398] T 60fb22f7cdeb4c979377537877845c16 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:39.558544 29105 tablet_bootstrap.cc:492] T 251e073eb0e44b02bb4ef9cb98aded4b P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:39.560693 29106 raft_consensus.cc:374] T 24d7b7f928ad446d8fe0d7cdc507c620 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:39.560914 29106 raft_consensus.cc:729] T 24d7b7f928ad446d8fe0d7cdc507c620 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:39.561350 29106 consensus_queue.cc:260] T 24d7b7f928ad446d8fe0d7cdc507c620 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:39.562495 29106 ts_tablet_manager.cc:1398] T 24d7b7f928ad446d8fe0d7cdc507c620 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240501 08:11:39.563273 29105 tablet_bootstrap.cc:654] T 251e073eb0e44b02bb4ef9cb98aded4b P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:39.572132 29105 tablet_bootstrap.cc:492] T 251e073eb0e44b02bb4ef9cb98aded4b P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:39.572505 29105 ts_tablet_manager.cc:1367] T 251e073eb0e44b02bb4ef9cb98aded4b P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20240501 08:11:39.573593 29105 raft_consensus.cc:348] T 251e073eb0e44b02bb4ef9cb98aded4b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:39.574046 29105 raft_consensus.cc:374] T 251e073eb0e44b02bb4ef9cb98aded4b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:39.574193 29105 raft_consensus.cc:729] T 251e073eb0e44b02bb4ef9cb98aded4b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:39.574563 29105 consensus_queue.cc:260] T 251e073eb0e44b02bb4ef9cb98aded4b P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:39.575608 29105 ts_tablet_manager.cc:1398] T 251e073eb0e44b02bb4ef9cb98aded4b P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:39.583895 29108 tablet_bootstrap.cc:492] T 251e073eb0e44b02bb4ef9cb98aded4b P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:39.584062 29108 ts_tablet_manager.cc:1367] T 251e073eb0e44b02bb4ef9cb98aded4b P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.054s user 0.011s sys 0.031s
I20240501 08:11:39.584769 29104 tablet_bootstrap.cc:492] T 251e073eb0e44b02bb4ef9cb98aded4b P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:39.584955 29104 ts_tablet_manager.cc:1367] T 251e073eb0e44b02bb4ef9cb98aded4b P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.052s user 0.019s sys 0.031s
I20240501 08:11:39.584852 29108 raft_consensus.cc:348] T 251e073eb0e44b02bb4ef9cb98aded4b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:39.585042 29108 raft_consensus.cc:374] T 251e073eb0e44b02bb4ef9cb98aded4b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:39.585160 29108 raft_consensus.cc:729] T 251e073eb0e44b02bb4ef9cb98aded4b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:39.585392 29108 consensus_queue.cc:260] T 251e073eb0e44b02bb4ef9cb98aded4b P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:39.585667 29104 raft_consensus.cc:348] T 251e073eb0e44b02bb4ef9cb98aded4b P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:39.585965 29104 raft_consensus.cc:374] T 251e073eb0e44b02bb4ef9cb98aded4b P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:39.586046 29108 ts_tablet_manager.cc:1398] T 251e073eb0e44b02bb4ef9cb98aded4b P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:11:39.586082 29104 raft_consensus.cc:729] T 251e073eb0e44b02bb4ef9cb98aded4b P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:39.586458 29108 tablet_bootstrap.cc:492] T 60fb22f7cdeb4c979377537877845c16 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:39.586359 29104 consensus_queue.cc:260] T 251e073eb0e44b02bb4ef9cb98aded4b P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:39.586918 29104 ts_tablet_manager.cc:1398] T 251e073eb0e44b02bb4ef9cb98aded4b P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:39.588182 29108 tablet_bootstrap.cc:654] T 60fb22f7cdeb4c979377537877845c16 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:39.592756 29108 tablet_bootstrap.cc:492] T 60fb22f7cdeb4c979377537877845c16 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:39.592927 29108 ts_tablet_manager.cc:1367] T 60fb22f7cdeb4c979377537877845c16 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 08:11:39.593591 29108 raft_consensus.cc:348] T 60fb22f7cdeb4c979377537877845c16 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:39.593811 29108 raft_consensus.cc:374] T 60fb22f7cdeb4c979377537877845c16 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:39.593894 29108 raft_consensus.cc:729] T 60fb22f7cdeb4c979377537877845c16 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:39.594146 29108 consensus_queue.cc:260] T 60fb22f7cdeb4c979377537877845c16 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:39.594736 29108 ts_tablet_manager.cc:1398] T 60fb22f7cdeb4c979377537877845c16 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:11:39.595055 29108 tablet_bootstrap.cc:492] T 24d7b7f928ad446d8fe0d7cdc507c620 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:39.596630 29108 tablet_bootstrap.cc:654] T 24d7b7f928ad446d8fe0d7cdc507c620 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:39.599653 29108 tablet_bootstrap.cc:492] T 24d7b7f928ad446d8fe0d7cdc507c620 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:39.599844 29108 ts_tablet_manager.cc:1367] T 24d7b7f928ad446d8fe0d7cdc507c620 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240501 08:11:39.600466 29108 raft_consensus.cc:348] T 24d7b7f928ad446d8fe0d7cdc507c620 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:39.600666 29108 raft_consensus.cc:374] T 24d7b7f928ad446d8fe0d7cdc507c620 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:39.600822 29108 raft_consensus.cc:729] T 24d7b7f928ad446d8fe0d7cdc507c620 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:39.601075 29108 consensus_queue.cc:260] T 24d7b7f928ad446d8fe0d7cdc507c620 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:39.601634 29108 ts_tablet_manager.cc:1398] T 24d7b7f928ad446d8fe0d7cdc507c620 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:11:39.650586 29030 raft_consensus.cc:482] T 251e073eb0e44b02bb4ef9cb98aded4b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:39.650821 29030 raft_consensus.cc:504] T 251e073eb0e44b02bb4ef9cb98aded4b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:39.651379 29030 leader_election.cc:288] T 251e073eb0e44b02bb4ef9cb98aded4b P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:39.652035 28020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "251e073eb0e44b02bb4ef9cb98aded4b" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:39.652089 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "251e073eb0e44b02bb4ef9cb98aded4b" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:11:39.652298 28020 raft_consensus.cc:2454] T 251e073eb0e44b02bb4ef9cb98aded4b P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:39.652315 27759 raft_consensus.cc:2454] T 251e073eb0e44b02bb4ef9cb98aded4b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:39.652750 28087 leader_election.cc:302] T 251e073eb0e44b02bb4ef9cb98aded4b P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:39.653061 29030 raft_consensus.cc:2781] T 251e073eb0e44b02bb4ef9cb98aded4b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:39.653148 29030 raft_consensus.cc:482] T 251e073eb0e44b02bb4ef9cb98aded4b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:39.653246 29030 raft_consensus.cc:3037] T 251e073eb0e44b02bb4ef9cb98aded4b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:39.655084 29030 raft_consensus.cc:504] T 251e073eb0e44b02bb4ef9cb98aded4b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:39.655602 29030 leader_election.cc:288] T 251e073eb0e44b02bb4ef9cb98aded4b P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:39.656196 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "251e073eb0e44b02bb4ef9cb98aded4b" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:39.656291 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "251e073eb0e44b02bb4ef9cb98aded4b" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:11:39.656378 28022 raft_consensus.cc:3037] T 251e073eb0e44b02bb4ef9cb98aded4b P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:39.656491 27759 raft_consensus.cc:3037] T 251e073eb0e44b02bb4ef9cb98aded4b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:39.658201 28022 raft_consensus.cc:2454] T 251e073eb0e44b02bb4ef9cb98aded4b P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:39.658417 27759 raft_consensus.cc:2454] T 251e073eb0e44b02bb4ef9cb98aded4b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:39.658597 28087 leader_election.cc:302] T 251e073eb0e44b02bb4ef9cb98aded4b P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:39.658912 29030 raft_consensus.cc:2781] T 251e073eb0e44b02bb4ef9cb98aded4b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:39.659060 29030 raft_consensus.cc:686] T 251e073eb0e44b02bb4ef9cb98aded4b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:11:39.659288 29030 consensus_queue.cc:237] T 251e073eb0e44b02bb4ef9cb98aded4b P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:39.662000 27618 catalog_manager.cc:5484] T 251e073eb0e44b02bb4ef9cb98aded4b P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:39.772606 29098 raft_consensus.cc:482] T 24d7b7f928ad446d8fe0d7cdc507c620 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:39.772825 29098 raft_consensus.cc:504] T 24d7b7f928ad446d8fe0d7cdc507c620 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:39.773571 29098 leader_election.cc:288] T 24d7b7f928ad446d8fe0d7cdc507c620 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:11:39.774053 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24d7b7f928ad446d8fe0d7cdc507c620" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:39.774159 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24d7b7f928ad446d8fe0d7cdc507c620" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:11:39.774297 28022 raft_consensus.cc:2454] T 24d7b7f928ad446d8fe0d7cdc507c620 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:11:39.774386 27889 raft_consensus.cc:2454] T 24d7b7f928ad446d8fe0d7cdc507c620 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:11:39.774781 27696 leader_election.cc:302] T 24d7b7f928ad446d8fe0d7cdc507c620 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:39.775051 29098 raft_consensus.cc:2781] T 24d7b7f928ad446d8fe0d7cdc507c620 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:39.775171 29098 raft_consensus.cc:482] T 24d7b7f928ad446d8fe0d7cdc507c620 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:39.775257 29098 raft_consensus.cc:3037] T 24d7b7f928ad446d8fe0d7cdc507c620 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:39.777077 29098 raft_consensus.cc:504] T 24d7b7f928ad446d8fe0d7cdc507c620 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:39.777693 29098 leader_election.cc:288] T 24d7b7f928ad446d8fe0d7cdc507c620 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Requested vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:11:39.778209 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24d7b7f928ad446d8fe0d7cdc507c620" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:39.778303 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24d7b7f928ad446d8fe0d7cdc507c620" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:11:39.778425 28022 raft_consensus.cc:3037] T 24d7b7f928ad446d8fe0d7cdc507c620 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:39.778506 27889 raft_consensus.cc:3037] T 24d7b7f928ad446d8fe0d7cdc507c620 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:39.780367 28022 raft_consensus.cc:2454] T 24d7b7f928ad446d8fe0d7cdc507c620 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:11:39.780459 27889 raft_consensus.cc:2454] T 24d7b7f928ad446d8fe0d7cdc507c620 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:11:39.780817 27696 leader_election.cc:302] T 24d7b7f928ad446d8fe0d7cdc507c620 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:39.781132 29098 raft_consensus.cc:2781] T 24d7b7f928ad446d8fe0d7cdc507c620 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:39.781553 29098 raft_consensus.cc:686] T 24d7b7f928ad446d8fe0d7cdc507c620 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:11:39.781810 29098 consensus_queue.cc:237] T 24d7b7f928ad446d8fe0d7cdc507c620 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:39.784408 27618 catalog_manager.cc:5484] T 24d7b7f928ad446d8fe0d7cdc507c620 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:39.814548 29098 raft_consensus.cc:482] T 60fb22f7cdeb4c979377537877845c16 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:39.814738 29098 raft_consensus.cc:504] T 60fb22f7cdeb4c979377537877845c16 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:39.815342 29098 leader_election.cc:288] T 60fb22f7cdeb4c979377537877845c16 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:11:39.815770 28152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60fb22f7cdeb4c979377537877845c16" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:11:39.815955 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60fb22f7cdeb4c979377537877845c16" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:11:39.816031 28152 raft_consensus.cc:2454] T 60fb22f7cdeb4c979377537877845c16 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:11:39.816234 27889 raft_consensus.cc:2454] T 60fb22f7cdeb4c979377537877845c16 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:11:39.816488 27694 leader_election.cc:302] T 60fb22f7cdeb4c979377537877845c16 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:11:39.816756 29098 raft_consensus.cc:2781] T 60fb22f7cdeb4c979377537877845c16 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:39.816864 29098 raft_consensus.cc:482] T 60fb22f7cdeb4c979377537877845c16 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:39.816985 29098 raft_consensus.cc:3037] T 60fb22f7cdeb4c979377537877845c16 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:39.818876 29098 raft_consensus.cc:504] T 60fb22f7cdeb4c979377537877845c16 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:39.819427 29098 leader_election.cc:288] T 60fb22f7cdeb4c979377537877845c16 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Requested vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:11:39.819870 28152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60fb22f7cdeb4c979377537877845c16" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:11:39.819870 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60fb22f7cdeb4c979377537877845c16" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:11:39.820089 27889 raft_consensus.cc:3037] T 60fb22f7cdeb4c979377537877845c16 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:39.820089 28152 raft_consensus.cc:3037] T 60fb22f7cdeb4c979377537877845c16 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:39.822757 27889 raft_consensus.cc:2454] T 60fb22f7cdeb4c979377537877845c16 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:11:39.822757 28152 raft_consensus.cc:2454] T 60fb22f7cdeb4c979377537877845c16 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:11:39.823169 27694 leader_election.cc:302] T 60fb22f7cdeb4c979377537877845c16 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:11:39.823446 29098 raft_consensus.cc:2781] T 60fb22f7cdeb4c979377537877845c16 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:39.823590 29098 raft_consensus.cc:686] T 60fb22f7cdeb4c979377537877845c16 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:11:39.823838 29098 consensus_queue.cc:237] T 60fb22f7cdeb4c979377537877845c16 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:39.826475 27618 catalog_manager.cc:5484] T 60fb22f7cdeb4c979377537877845c16 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:40.097579 29099 consensus_queue.cc:1035] T 251e073eb0e44b02bb4ef9cb98aded4b P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:40.106040 29099 consensus_queue.cc:1035] T 251e073eb0e44b02bb4ef9cb98aded4b P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:40.177800 29122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:40.178206 29122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:40.185802 29122 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 08:11:40.302441 29098 consensus_queue.cc:1035] T 60fb22f7cdeb4c979377537877845c16 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:40.317127 29119 consensus_queue.cc:1035] T 60fb22f7cdeb4c979377537877845c16 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:40.337168 27889 raft_consensus.cc:1261] T 24d7b7f928ad446d8fe0d7cdc507c620 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:11:40.338032 28022 raft_consensus.cc:1261] T 24d7b7f928ad446d8fe0d7cdc507c620 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:11:40.338665 29119 consensus_queue.cc:1035] T 24d7b7f928ad446d8fe0d7cdc507c620 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:40.339443 29119 consensus_queue.cc:1035] T 24d7b7f928ad446d8fe0d7cdc507c620 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:40.389513 29156 mvcc.cc:205] Tried to move back new op lower bound from 7022801306971750400 to 7022801304708874240. Current Snapshot: MvccSnapshot[applied={T|T < 7022801306971750400}]
I20240501 08:11:40.414327 29125 mvcc.cc:205] Tried to move back new op lower bound from 7022801307240091648 to 7022801304708874240. Current Snapshot: MvccSnapshot[applied={T|T < 7022801306971750400}]
I20240501 08:11:40.419270 29129 mvcc.cc:205] Tried to move back new op lower bound from 7022801306971750400 to 7022801304708874240. Current Snapshot: MvccSnapshot[applied={T|T < 7022801306971750400}]
INSERT report
rows total: 2000
time total: 237.095 ms
time per row: 0.118547 ms
I20240501 08:11:40.586395 27614 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:40.595382 28001 tablet_service.cc:1467] Processing CreateTablet for tablet bc5778711a4c479881088af790f5b7d5 (DEFAULT_TABLE table=rebalance_test_table_5 [id=b19758166c02438c9fc4cfa835acc028]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:40.595686 27872 tablet_service.cc:1467] Processing CreateTablet for tablet bc5778711a4c479881088af790f5b7d5 (DEFAULT_TABLE table=rebalance_test_table_5 [id=b19758166c02438c9fc4cfa835acc028]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:40.595831 28001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc5778711a4c479881088af790f5b7d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:40.596128 27872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc5778711a4c479881088af790f5b7d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:40.597209 28130 tablet_service.cc:1467] Processing CreateTablet for tablet bc5778711a4c479881088af790f5b7d5 (DEFAULT_TABLE table=rebalance_test_table_5 [id=b19758166c02438c9fc4cfa835acc028]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:40.597600 28130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc5778711a4c479881088af790f5b7d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:40.598760 27870 tablet_service.cc:1467] Processing CreateTablet for tablet e3d07789edc74dec975e35e9d94cb0ef (DEFAULT_TABLE table=rebalance_test_table_5 [id=b19758166c02438c9fc4cfa835acc028]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:40.599130 27870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3d07789edc74dec975e35e9d94cb0ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:40.600684 27871 tablet_service.cc:1467] Processing CreateTablet for tablet 05b2485fa01a4706a85ebc150306637f (DEFAULT_TABLE table=rebalance_test_table_5 [id=b19758166c02438c9fc4cfa835acc028]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:40.601040 27871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05b2485fa01a4706a85ebc150306637f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:40.603395 28131 tablet_service.cc:1467] Processing CreateTablet for tablet 05b2485fa01a4706a85ebc150306637f (DEFAULT_TABLE table=rebalance_test_table_5 [id=b19758166c02438c9fc4cfa835acc028]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:40.603482 28132 tablet_service.cc:1467] Processing CreateTablet for tablet e3d07789edc74dec975e35e9d94cb0ef (DEFAULT_TABLE table=rebalance_test_table_5 [id=b19758166c02438c9fc4cfa835acc028]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:40.603783 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05b2485fa01a4706a85ebc150306637f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:40.604012 28132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3d07789edc74dec975e35e9d94cb0ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:40.606632 27739 tablet_service.cc:1467] Processing CreateTablet for tablet e3d07789edc74dec975e35e9d94cb0ef (DEFAULT_TABLE table=rebalance_test_table_5 [id=b19758166c02438c9fc4cfa835acc028]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:40.607115 27739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3d07789edc74dec975e35e9d94cb0ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:40.611737 28000 tablet_service.cc:1467] Processing CreateTablet for tablet 05b2485fa01a4706a85ebc150306637f (DEFAULT_TABLE table=rebalance_test_table_5 [id=b19758166c02438c9fc4cfa835acc028]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:40.612288 28000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05b2485fa01a4706a85ebc150306637f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:40.612567 29189 tablet_bootstrap.cc:492] T bc5778711a4c479881088af790f5b7d5 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:40.622928 29189 tablet_bootstrap.cc:654] T bc5778711a4c479881088af790f5b7d5 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:40.628060 29190 tablet_bootstrap.cc:492] T e3d07789edc74dec975e35e9d94cb0ef P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:40.628386 29189 tablet_bootstrap.cc:492] T bc5778711a4c479881088af790f5b7d5 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:40.628544 29189 ts_tablet_manager.cc:1367] T bc5778711a4c479881088af790f5b7d5 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.004s
I20240501 08:11:40.630026 29189 raft_consensus.cc:348] T bc5778711a4c479881088af790f5b7d5 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:40.630227 29191 tablet_bootstrap.cc:492] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:40.630466 29189 raft_consensus.cc:374] T bc5778711a4c479881088af790f5b7d5 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:40.630590 29193 tablet_bootstrap.cc:492] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:40.630630 29189 raft_consensus.cc:729] T bc5778711a4c479881088af790f5b7d5 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:40.630995 29189 consensus_queue.cc:260] T bc5778711a4c479881088af790f5b7d5 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:40.631692 29189 ts_tablet_manager.cc:1398] T bc5778711a4c479881088af790f5b7d5 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 08:11:40.632171 29190 tablet_bootstrap.cc:654] T e3d07789edc74dec975e35e9d94cb0ef P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:40.632601 29193 tablet_bootstrap.cc:654] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:40.632634 29189 tablet_bootstrap.cc:492] T 05b2485fa01a4706a85ebc150306637f P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:40.638500 29191 tablet_bootstrap.cc:654] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:40.639719 29189 tablet_bootstrap.cc:654] T 05b2485fa01a4706a85ebc150306637f P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:40.641386 29193 tablet_bootstrap.cc:492] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:40.641655 29193 ts_tablet_manager.cc:1367] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240501 08:11:40.642850 29193 raft_consensus.cc:348] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:40.643229 29193 raft_consensus.cc:374] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:40.643386 29193 raft_consensus.cc:729] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:40.643790 29193 consensus_queue.cc:260] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:40.644773 29193 ts_tablet_manager.cc:1398] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:11:40.646670 29191 tablet_bootstrap.cc:492] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:40.646878 29191 ts_tablet_manager.cc:1367] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.006s
I20240501 08:11:40.647603 29190 tablet_bootstrap.cc:492] T e3d07789edc74dec975e35e9d94cb0ef P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:40.647781 29190 ts_tablet_manager.cc:1367] T e3d07789edc74dec975e35e9d94cb0ef P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.000s
I20240501 08:11:40.647817 29191 raft_consensus.cc:348] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:40.647991 29179 raft_consensus.cc:482] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:40.648180 29191 raft_consensus.cc:374] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:40.648145 29179 raft_consensus.cc:504] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:40.648327 29191 raft_consensus.cc:729] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:40.648557 29191 consensus_queue.cc:260] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:40.648634 29190 raft_consensus.cc:348] T e3d07789edc74dec975e35e9d94cb0ef P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:40.648840 29190 raft_consensus.cc:374] T e3d07789edc74dec975e35e9d94cb0ef P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:40.648874 29179 leader_election.cc:288] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:11:40.648921 29190 raft_consensus.cc:729] T e3d07789edc74dec975e35e9d94cb0ef P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:40.649173 29190 consensus_queue.cc:260] T e3d07789edc74dec975e35e9d94cb0ef P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:40.649863 29190 ts_tablet_manager.cc:1398] T e3d07789edc74dec975e35e9d94cb0ef P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:40.650205 29190 tablet_bootstrap.cc:492] T bc5778711a4c479881088af790f5b7d5 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:40.650516 28152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d07789edc74dec975e35e9d94cb0ef" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:11:40.651181 29189 tablet_bootstrap.cc:492] T 05b2485fa01a4706a85ebc150306637f P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:40.651348 29189 ts_tablet_manager.cc:1367] T 05b2485fa01a4706a85ebc150306637f P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
W20240501 08:11:40.651559 27694 leader_election.cc:341] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:11:40.651839 29190 tablet_bootstrap.cc:654] T bc5778711a4c479881088af790f5b7d5 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:40.652119 29189 raft_consensus.cc:348] T 05b2485fa01a4706a85ebc150306637f P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:40.652350 29189 raft_consensus.cc:374] T 05b2485fa01a4706a85ebc150306637f P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:40.652422 29189 raft_consensus.cc:729] T 05b2485fa01a4706a85ebc150306637f P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:40.649794 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d07789edc74dec975e35e9d94cb0ef" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:11:40.652637 27889 raft_consensus.cc:2454] T e3d07789edc74dec975e35e9d94cb0ef P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:11:40.652674 29189 consensus_queue.cc:260] T 05b2485fa01a4706a85ebc150306637f P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:40.653079 27697 leader_election.cc:302] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, f1525cbc82a24d13b65f136695c903af; no voters: 91d11eae5ef6432ca86eb8ed446b6ea6
I20240501 08:11:40.654489 29189 ts_tablet_manager.cc:1398] T 05b2485fa01a4706a85ebc150306637f P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:40.649223 29191 ts_tablet_manager.cc:1398] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:11:40.655050 29191 tablet_bootstrap.cc:492] T e3d07789edc74dec975e35e9d94cb0ef P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:40.657181 29179 raft_consensus.cc:2781] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:40.657297 29179 raft_consensus.cc:482] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:40.657374 29179 raft_consensus.cc:3037] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:40.658593 29191 tablet_bootstrap.cc:654] T e3d07789edc74dec975e35e9d94cb0ef P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:40.660238 29179 raft_consensus.cc:504] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:40.661350 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d07789edc74dec975e35e9d94cb0ef" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:11:40.661579 27889 raft_consensus.cc:3037] T e3d07789edc74dec975e35e9d94cb0ef P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:40.662043 29190 tablet_bootstrap.cc:492] T bc5778711a4c479881088af790f5b7d5 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:40.662231 29190 ts_tablet_manager.cc:1367] T bc5778711a4c479881088af790f5b7d5 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.007s
I20240501 08:11:40.663507 29190 raft_consensus.cc:348] T bc5778711a4c479881088af790f5b7d5 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:40.663776 29190 raft_consensus.cc:374] T bc5778711a4c479881088af790f5b7d5 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:40.663870 29190 raft_consensus.cc:729] T bc5778711a4c479881088af790f5b7d5 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:40.664192 29190 consensus_queue.cc:260] T bc5778711a4c479881088af790f5b7d5 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:40.665377 27889 raft_consensus.cc:2454] T e3d07789edc74dec975e35e9d94cb0ef P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:11:40.665657 29191 tablet_bootstrap.cc:492] T e3d07789edc74dec975e35e9d94cb0ef P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:40.665867 29191 ts_tablet_manager.cc:1367] T e3d07789edc74dec975e35e9d94cb0ef P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240501 08:11:40.666960 29179 leader_election.cc:288] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Requested vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:11:40.667198 29191 raft_consensus.cc:348] T e3d07789edc74dec975e35e9d94cb0ef P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:40.667474 29191 raft_consensus.cc:374] T e3d07789edc74dec975e35e9d94cb0ef P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:40.667376 28152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d07789edc74dec975e35e9d94cb0ef" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:11:40.667614 27697 leader_election.cc:302] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:40.665452 29190 ts_tablet_manager.cc:1398] T bc5778711a4c479881088af790f5b7d5 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:40.667948 29179 raft_consensus.cc:2781] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:40.668154 29190 tablet_bootstrap.cc:492] T 05b2485fa01a4706a85ebc150306637f P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:40.668148 29179 raft_consensus.cc:686] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:11:40.667636 29191 raft_consensus.cc:729] T e3d07789edc74dec975e35e9d94cb0ef P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:40.668696 29191 consensus_queue.cc:260] T e3d07789edc74dec975e35e9d94cb0ef P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:40.668980 28152 raft_consensus.cc:3037] T e3d07789edc74dec975e35e9d94cb0ef P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:40.670243 29190 tablet_bootstrap.cc:654] T 05b2485fa01a4706a85ebc150306637f P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:40.668465 29179 consensus_queue.cc:237] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:40.671926 28152 raft_consensus.cc:2454] T e3d07789edc74dec975e35e9d94cb0ef P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:11:40.674422 29190 tablet_bootstrap.cc:492] T 05b2485fa01a4706a85ebc150306637f P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:40.674573 29191 ts_tablet_manager.cc:1398] T e3d07789edc74dec975e35e9d94cb0ef P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240501 08:11:40.674599 29190 ts_tablet_manager.cc:1367] T 05b2485fa01a4706a85ebc150306637f P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240501 08:11:40.674903 29191 tablet_bootstrap.cc:492] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:40.675460 29190 raft_consensus.cc:348] T 05b2485fa01a4706a85ebc150306637f P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:40.675752 29190 raft_consensus.cc:374] T 05b2485fa01a4706a85ebc150306637f P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:40.675863 29190 raft_consensus.cc:729] T 05b2485fa01a4706a85ebc150306637f P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:40.676208 29190 consensus_queue.cc:260] T 05b2485fa01a4706a85ebc150306637f P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:40.676909 29190 ts_tablet_manager.cc:1398] T 05b2485fa01a4706a85ebc150306637f P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:40.679204 29191 tablet_bootstrap.cc:654] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:40.682471 29191 tablet_bootstrap.cc:492] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:40.682646 29191 ts_tablet_manager.cc:1367] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20240501 08:11:40.683255 29191 raft_consensus.cc:348] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:40.683411 29191 raft_consensus.cc:374] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:40.683279 27614 catalog_manager.cc:5484] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:40.683506 29191 raft_consensus.cc:729] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:40.683701 29191 consensus_queue.cc:260] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:40.684180 29191 ts_tablet_manager.cc:1398] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:11:40.794109 29030 raft_consensus.cc:482] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:40.794340 29030 raft_consensus.cc:504] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:40.795106 29030 leader_election.cc:288] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:11:40.795514 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc5778711a4c479881088af790f5b7d5" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:40.795619 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc5778711a4c479881088af790f5b7d5" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:11:40.795722 28022 raft_consensus.cc:2454] T bc5778711a4c479881088af790f5b7d5 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:40.795877 27889 raft_consensus.cc:2454] T bc5778711a4c479881088af790f5b7d5 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:40.796200 28087 leader_election.cc:302] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:40.796463 29030 raft_consensus.cc:2781] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:40.796574 29030 raft_consensus.cc:482] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:40.796645 29030 raft_consensus.cc:3037] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:40.798501 29030 raft_consensus.cc:504] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:40.799084 29030 leader_election.cc:288] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:11:40.799482 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc5778711a4c479881088af790f5b7d5" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:40.799588 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc5778711a4c479881088af790f5b7d5" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:11:40.799681 28022 raft_consensus.cc:3037] T bc5778711a4c479881088af790f5b7d5 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:40.799787 27889 raft_consensus.cc:3037] T bc5778711a4c479881088af790f5b7d5 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:40.801636 28022 raft_consensus.cc:2454] T bc5778711a4c479881088af790f5b7d5 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:40.801682 27889 raft_consensus.cc:2454] T bc5778711a4c479881088af790f5b7d5 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:40.802042 28087 leader_election.cc:302] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:40.802345 29030 raft_consensus.cc:2781] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:40.802484 29030 raft_consensus.cc:686] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:11:40.802748 29030 consensus_queue.cc:237] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:40.805568 27614 catalog_manager.cc:5484] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:40.819173 29030 raft_consensus.cc:482] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:40.819357 29030 raft_consensus.cc:504] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:40.820058 29030 leader_election.cc:288] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:11:40.820432 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05b2485fa01a4706a85ebc150306637f" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:11:40.820669 27889 raft_consensus.cc:2454] T 05b2485fa01a4706a85ebc150306637f P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:40.820601 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05b2485fa01a4706a85ebc150306637f" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:40.820859 28022 raft_consensus.cc:2454] T 05b2485fa01a4706a85ebc150306637f P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:40.821061 28088 leader_election.cc:302] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:40.821323 29030 raft_consensus.cc:2781] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:40.821398 29030 raft_consensus.cc:482] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:40.821491 29030 raft_consensus.cc:3037] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:40.847826 29030 raft_consensus.cc:504] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:40.848505 29030 leader_election.cc:288] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:11:40.849074 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05b2485fa01a4706a85ebc150306637f" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:40.849076 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05b2485fa01a4706a85ebc150306637f" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:11:40.849339 28022 raft_consensus.cc:3037] T 05b2485fa01a4706a85ebc150306637f P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:40.849368 27889 raft_consensus.cc:3037] T 05b2485fa01a4706a85ebc150306637f P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:40.851444 27889 raft_consensus.cc:2454] T 05b2485fa01a4706a85ebc150306637f P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:40.851445 28022 raft_consensus.cc:2454] T 05b2485fa01a4706a85ebc150306637f P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:40.851944 28088 leader_election.cc:302] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:40.852265 29030 raft_consensus.cc:2781] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:40.852421 29030 raft_consensus.cc:686] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:11:40.852684 29030 consensus_queue.cc:237] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:40.855321 27614 catalog_manager.cc:5484] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:41.066596 29179 consensus_queue.cc:1035] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:41.077441 29179 consensus_queue.cc:1035] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:41.222982 29205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:41.223312 29205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:41.230773 29205 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240501 08:11:41.281020 29030 consensus_queue.cc:1035] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:41.286811 29126 consensus_queue.cc:1035] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:41.312546 29126 consensus_queue.cc:1035] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:41.318297 29126 consensus_queue.cc:1035] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 248.331 ms
time per row: 0.124166 ms
I20240501 08:11:41.624677 27616 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:41.634357 27870 tablet_service.cc:1467] Processing CreateTablet for tablet 20425913301a4d8abb9621cc7db0e3e7 (DEFAULT_TABLE table=rebalance_test_table_6 [id=bb586b867d1742378689aab16d2baa26]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:41.634544 28001 tablet_service.cc:1467] Processing CreateTablet for tablet 20425913301a4d8abb9621cc7db0e3e7 (DEFAULT_TABLE table=rebalance_test_table_6 [id=bb586b867d1742378689aab16d2baa26]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:41.634830 27870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20425913301a4d8abb9621cc7db0e3e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:41.634761 27739 tablet_service.cc:1467] Processing CreateTablet for tablet 20425913301a4d8abb9621cc7db0e3e7 (DEFAULT_TABLE table=rebalance_test_table_6 [id=bb586b867d1742378689aab16d2baa26]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:41.634929 28001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20425913301a4d8abb9621cc7db0e3e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:41.634945 28131 tablet_service.cc:1467] Processing CreateTablet for tablet e258bb4e672f4795a3b6739f8636716e (DEFAULT_TABLE table=rebalance_test_table_6 [id=bb586b867d1742378689aab16d2baa26]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:41.635186 27739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20425913301a4d8abb9621cc7db0e3e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:41.635322 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e258bb4e672f4795a3b6739f8636716e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:41.639206 27740 tablet_service.cc:1467] Processing CreateTablet for tablet e258bb4e672f4795a3b6739f8636716e (DEFAULT_TABLE table=rebalance_test_table_6 [id=bb586b867d1742378689aab16d2baa26]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:41.639633 27740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e258bb4e672f4795a3b6739f8636716e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:41.648936 29269 tablet_bootstrap.cc:492] T e258bb4e672f4795a3b6739f8636716e P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:41.650420 28000 tablet_service.cc:1467] Processing CreateTablet for tablet e258bb4e672f4795a3b6739f8636716e (DEFAULT_TABLE table=rebalance_test_table_6 [id=bb586b867d1742378689aab16d2baa26]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:41.650919 28000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e258bb4e672f4795a3b6739f8636716e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:41.651526 29269 tablet_bootstrap.cc:654] T e258bb4e672f4795a3b6739f8636716e P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:41.653226 29268 tablet_bootstrap.cc:492] T 20425913301a4d8abb9621cc7db0e3e7 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:41.654377 28002 tablet_service.cc:1467] Processing CreateTablet for tablet 0d9538e4316a403e805ba5d06e6585d8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=bb586b867d1742378689aab16d2baa26]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:41.654475 27872 tablet_service.cc:1467] Processing CreateTablet for tablet 0d9538e4316a403e805ba5d06e6585d8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=bb586b867d1742378689aab16d2baa26]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:41.654923 27872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d9538e4316a403e805ba5d06e6585d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:41.654927 28002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d9538e4316a403e805ba5d06e6585d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:41.654995 29268 tablet_bootstrap.cc:654] T 20425913301a4d8abb9621cc7db0e3e7 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:41.659150 27740 tablet_service.cc:1467] Processing CreateTablet for tablet 0d9538e4316a403e805ba5d06e6585d8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=bb586b867d1742378689aab16d2baa26]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:41.659803 27740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d9538e4316a403e805ba5d06e6585d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:41.660074 29272 tablet_bootstrap.cc:492] T 20425913301a4d8abb9621cc7db0e3e7 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:41.663098 29272 tablet_bootstrap.cc:654] T 20425913301a4d8abb9621cc7db0e3e7 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:41.664876 29268 tablet_bootstrap.cc:492] T 20425913301a4d8abb9621cc7db0e3e7 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:41.665084 29268 ts_tablet_manager.cc:1367] T 20425913301a4d8abb9621cc7db0e3e7 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20240501 08:11:41.666385 29268 raft_consensus.cc:348] T 20425913301a4d8abb9621cc7db0e3e7 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:41.666694 29268 raft_consensus.cc:374] T 20425913301a4d8abb9621cc7db0e3e7 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:41.666816 29268 raft_consensus.cc:729] T 20425913301a4d8abb9621cc7db0e3e7 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:41.667189 29268 consensus_queue.cc:260] T 20425913301a4d8abb9621cc7db0e3e7 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:41.668855 29268 ts_tablet_manager.cc:1398] T 20425913301a4d8abb9621cc7db0e3e7 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:11:41.669166 29268 tablet_bootstrap.cc:492] T e258bb4e672f4795a3b6739f8636716e P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:41.669335 29267 tablet_bootstrap.cc:492] T 20425913301a4d8abb9621cc7db0e3e7 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:41.670666 29269 tablet_bootstrap.cc:492] T e258bb4e672f4795a3b6739f8636716e P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:41.670848 29269 ts_tablet_manager.cc:1367] T e258bb4e672f4795a3b6739f8636716e P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.000s
I20240501 08:11:41.671545 29267 tablet_bootstrap.cc:654] T 20425913301a4d8abb9621cc7db0e3e7 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:41.671680 29269 raft_consensus.cc:348] T e258bb4e672f4795a3b6739f8636716e P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:41.672693 29268 tablet_bootstrap.cc:654] T e258bb4e672f4795a3b6739f8636716e P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:41.672842 29269 raft_consensus.cc:374] T e258bb4e672f4795a3b6739f8636716e P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:41.672942 29269 raft_consensus.cc:729] T e258bb4e672f4795a3b6739f8636716e P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:41.673259 29269 consensus_queue.cc:260] T e258bb4e672f4795a3b6739f8636716e P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:41.673914 29269 ts_tablet_manager.cc:1398] T e258bb4e672f4795a3b6739f8636716e P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:41.674083 29267 tablet_bootstrap.cc:492] T 20425913301a4d8abb9621cc7db0e3e7 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:41.674226 29267 ts_tablet_manager.cc:1367] T 20425913301a4d8abb9621cc7db0e3e7 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 08:11:41.675138 29267 raft_consensus.cc:348] T 20425913301a4d8abb9621cc7db0e3e7 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:41.675323 29267 raft_consensus.cc:374] T 20425913301a4d8abb9621cc7db0e3e7 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:41.675431 29267 raft_consensus.cc:729] T 20425913301a4d8abb9621cc7db0e3e7 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:41.675721 29267 consensus_queue.cc:260] T 20425913301a4d8abb9621cc7db0e3e7 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:41.675971 29268 tablet_bootstrap.cc:492] T e258bb4e672f4795a3b6739f8636716e P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:41.676129 29268 ts_tablet_manager.cc:1367] T e258bb4e672f4795a3b6739f8636716e P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240501 08:11:41.676496 29267 ts_tablet_manager.cc:1398] T 20425913301a4d8abb9621cc7db0e3e7 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:41.676927 29268 raft_consensus.cc:348] T e258bb4e672f4795a3b6739f8636716e P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:41.677162 29268 raft_consensus.cc:374] T e258bb4e672f4795a3b6739f8636716e P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:41.677242 29268 raft_consensus.cc:729] T e258bb4e672f4795a3b6739f8636716e P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:41.677575 29267 tablet_bootstrap.cc:492] T 0d9538e4316a403e805ba5d06e6585d8 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:41.677503 29268 consensus_queue.cc:260] T e258bb4e672f4795a3b6739f8636716e P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:41.678174 29268 ts_tablet_manager.cc:1398] T e258bb4e672f4795a3b6739f8636716e P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:41.678447 29268 tablet_bootstrap.cc:492] T 0d9538e4316a403e805ba5d06e6585d8 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:41.679383 29267 tablet_bootstrap.cc:654] T 0d9538e4316a403e805ba5d06e6585d8 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:41.680048 29268 tablet_bootstrap.cc:654] T 0d9538e4316a403e805ba5d06e6585d8 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:41.685110 29268 tablet_bootstrap.cc:492] T 0d9538e4316a403e805ba5d06e6585d8 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:41.685271 29268 ts_tablet_manager.cc:1367] T 0d9538e4316a403e805ba5d06e6585d8 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20240501 08:11:41.685428 29267 tablet_bootstrap.cc:492] T 0d9538e4316a403e805ba5d06e6585d8 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:41.685606 29267 ts_tablet_manager.cc:1367] T 0d9538e4316a403e805ba5d06e6585d8 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240501 08:11:41.686046 29268 raft_consensus.cc:348] T 0d9538e4316a403e805ba5d06e6585d8 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:41.686269 29268 raft_consensus.cc:374] T 0d9538e4316a403e805ba5d06e6585d8 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:41.686352 29268 raft_consensus.cc:729] T 0d9538e4316a403e805ba5d06e6585d8 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:41.686396 29267 raft_consensus.cc:348] T 0d9538e4316a403e805ba5d06e6585d8 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:41.686583 29267 raft_consensus.cc:374] T 0d9538e4316a403e805ba5d06e6585d8 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:41.686643 29267 raft_consensus.cc:729] T 0d9538e4316a403e805ba5d06e6585d8 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:41.686576 29268 consensus_queue.cc:260] T 0d9538e4316a403e805ba5d06e6585d8 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:41.687132 29268 ts_tablet_manager.cc:1398] T 0d9538e4316a403e805ba5d06e6585d8 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:41.686874 29267 consensus_queue.cc:260] T 0d9538e4316a403e805ba5d06e6585d8 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:41.689319 29267 ts_tablet_manager.cc:1398] T 0d9538e4316a403e805ba5d06e6585d8 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:11:41.694612 29272 tablet_bootstrap.cc:492] T 20425913301a4d8abb9621cc7db0e3e7 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:41.694794 29272 ts_tablet_manager.cc:1367] T 20425913301a4d8abb9621cc7db0e3e7 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.003s
I20240501 08:11:41.695603 29272 raft_consensus.cc:348] T 20425913301a4d8abb9621cc7db0e3e7 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:41.695813 29272 raft_consensus.cc:374] T 20425913301a4d8abb9621cc7db0e3e7 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:41.695917 29272 raft_consensus.cc:729] T 20425913301a4d8abb9621cc7db0e3e7 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:41.696125 29272 consensus_queue.cc:260] T 20425913301a4d8abb9621cc7db0e3e7 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:41.696568 29272 ts_tablet_manager.cc:1398] T 20425913301a4d8abb9621cc7db0e3e7 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:11:41.696811 29272 tablet_bootstrap.cc:492] T e258bb4e672f4795a3b6739f8636716e P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:41.698436 29272 tablet_bootstrap.cc:654] T e258bb4e672f4795a3b6739f8636716e P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:41.700295 29272 tablet_bootstrap.cc:492] T e258bb4e672f4795a3b6739f8636716e P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:41.700445 29272 ts_tablet_manager.cc:1367] T e258bb4e672f4795a3b6739f8636716e P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:11:41.701169 29272 raft_consensus.cc:348] T e258bb4e672f4795a3b6739f8636716e P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:41.701351 29272 raft_consensus.cc:374] T e258bb4e672f4795a3b6739f8636716e P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:41.701426 29272 raft_consensus.cc:729] T e258bb4e672f4795a3b6739f8636716e P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:41.701635 29272 consensus_queue.cc:260] T e258bb4e672f4795a3b6739f8636716e P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:41.702142 29272 ts_tablet_manager.cc:1398] T e258bb4e672f4795a3b6739f8636716e P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:41.702518 29272 tablet_bootstrap.cc:492] T 0d9538e4316a403e805ba5d06e6585d8 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:41.703899 29272 tablet_bootstrap.cc:654] T 0d9538e4316a403e805ba5d06e6585d8 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:41.706830 29272 tablet_bootstrap.cc:492] T 0d9538e4316a403e805ba5d06e6585d8 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:41.706981 29272 ts_tablet_manager.cc:1367] T 0d9538e4316a403e805ba5d06e6585d8 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 08:11:41.707553 29272 raft_consensus.cc:348] T 0d9538e4316a403e805ba5d06e6585d8 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:41.707713 29272 raft_consensus.cc:374] T 0d9538e4316a403e805ba5d06e6585d8 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:41.707804 29272 raft_consensus.cc:729] T 0d9538e4316a403e805ba5d06e6585d8 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:41.708030 29272 consensus_queue.cc:260] T 0d9538e4316a403e805ba5d06e6585d8 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:41.709077 29272 ts_tablet_manager.cc:1398] T 0d9538e4316a403e805ba5d06e6585d8 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:11:41.770339 29171 raft_consensus.cc:482] T e258bb4e672f4795a3b6739f8636716e P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:41.770536 29171 raft_consensus.cc:504] T e258bb4e672f4795a3b6739f8636716e P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:41.771121 29171 leader_election.cc:288] T e258bb4e672f4795a3b6739f8636716e P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:11:41.771616 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e258bb4e672f4795a3b6739f8636716e" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:41.771651 28152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e258bb4e672f4795a3b6739f8636716e" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:11:41.771823 28022 raft_consensus.cc:2454] T e258bb4e672f4795a3b6739f8636716e P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:11:41.771858 28152 raft_consensus.cc:2454] T e258bb4e672f4795a3b6739f8636716e P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:11:41.772233 27696 leader_election.cc:302] T e258bb4e672f4795a3b6739f8636716e P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:41.772538 29171 raft_consensus.cc:2781] T e258bb4e672f4795a3b6739f8636716e P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:41.772673 29171 raft_consensus.cc:482] T e258bb4e672f4795a3b6739f8636716e P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:41.772763 29171 raft_consensus.cc:3037] T e258bb4e672f4795a3b6739f8636716e P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:41.774863 29171 raft_consensus.cc:504] T e258bb4e672f4795a3b6739f8636716e P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:41.775357 29171 leader_election.cc:288] T e258bb4e672f4795a3b6739f8636716e P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Requested vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:11:41.775810 28152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e258bb4e672f4795a3b6739f8636716e" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:11:41.775871 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e258bb4e672f4795a3b6739f8636716e" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:41.775976 28152 raft_consensus.cc:3037] T e258bb4e672f4795a3b6739f8636716e P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:41.776090 28022 raft_consensus.cc:3037] T e258bb4e672f4795a3b6739f8636716e P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:41.778084 28152 raft_consensus.cc:2454] T e258bb4e672f4795a3b6739f8636716e P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:11:41.778172 28022 raft_consensus.cc:2454] T e258bb4e672f4795a3b6739f8636716e P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:11:41.778519 27694 leader_election.cc:302] T e258bb4e672f4795a3b6739f8636716e P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:11:41.778837 29171 raft_consensus.cc:2781] T e258bb4e672f4795a3b6739f8636716e P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:41.778967 29171 raft_consensus.cc:686] T e258bb4e672f4795a3b6739f8636716e P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:11:41.779261 29171 consensus_queue.cc:237] T e258bb4e672f4795a3b6739f8636716e P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:41.782133 27615 catalog_manager.cc:5484] T e258bb4e672f4795a3b6739f8636716e P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:41.815183 29232 raft_consensus.cc:482] T 20425913301a4d8abb9621cc7db0e3e7 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:41.815382 29232 raft_consensus.cc:504] T 20425913301a4d8abb9621cc7db0e3e7 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:41.816190 29232 leader_election.cc:288] T 20425913301a4d8abb9621cc7db0e3e7 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:11:41.816792 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20425913301a4d8abb9621cc7db0e3e7" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:11:41.817037 27890 raft_consensus.cc:2454] T 20425913301a4d8abb9621cc7db0e3e7 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 0.
I20240501 08:11:41.817520 27958 leader_election.cc:302] T 20425913301a4d8abb9621cc7db0e3e7 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aad81047009a4ffcbc450b60030583b5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:41.817888 29232 raft_consensus.cc:2781] T 20425913301a4d8abb9621cc7db0e3e7 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:41.818065 29232 raft_consensus.cc:482] T 20425913301a4d8abb9621cc7db0e3e7 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:41.818173 29232 raft_consensus.cc:3037] T 20425913301a4d8abb9621cc7db0e3e7 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:41.820489 29232 raft_consensus.cc:504] T 20425913301a4d8abb9621cc7db0e3e7 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:41.821116 29232 leader_election.cc:288] T 20425913301a4d8abb9621cc7db0e3e7 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Requested vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:11:41.821571 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20425913301a4d8abb9621cc7db0e3e7" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:11:41.821764 27890 raft_consensus.cc:3037] T 20425913301a4d8abb9621cc7db0e3e7 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:41.823119 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20425913301a4d8abb9621cc7db0e3e7" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:11:41.823189 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20425913301a4d8abb9621cc7db0e3e7" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:11:41.823395 27759 raft_consensus.cc:2454] T 20425913301a4d8abb9621cc7db0e3e7 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 0.
I20240501 08:11:41.824258 27890 raft_consensus.cc:2454] T 20425913301a4d8abb9621cc7db0e3e7 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 1.
I20240501 08:11:41.824579 27958 leader_election.cc:302] T 20425913301a4d8abb9621cc7db0e3e7 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: aad81047009a4ffcbc450b60030583b5, f1525cbc82a24d13b65f136695c903af; no voters: 6331f77dae9d452184e8cd0cbf5f1b43
I20240501 08:11:41.824903 29232 raft_consensus.cc:2781] T 20425913301a4d8abb9621cc7db0e3e7 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:41.825280 29232 raft_consensus.cc:686] T 20425913301a4d8abb9621cc7db0e3e7 P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Becoming Leader. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Running, Role: LEADER
I20240501 08:11:41.825745 29232 consensus_queue.cc:237] T 20425913301a4d8abb9621cc7db0e3e7 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:41.829392 27615 catalog_manager.cc:5484] T 20425913301a4d8abb9621cc7db0e3e7 P aad81047009a4ffcbc450b60030583b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to aad81047009a4ffcbc450b60030583b5 (127.21.84.131). New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:41.978758 29171 raft_consensus.cc:482] T 0d9538e4316a403e805ba5d06e6585d8 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:41.978950 29171 raft_consensus.cc:504] T 0d9538e4316a403e805ba5d06e6585d8 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:41.979503 29171 leader_election.cc:288] T 0d9538e4316a403e805ba5d06e6585d8 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:11:41.980080 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9538e4316a403e805ba5d06e6585d8" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:41.980057 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9538e4316a403e805ba5d06e6585d8" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:11:41.980319 28022 raft_consensus.cc:2454] T 0d9538e4316a403e805ba5d06e6585d8 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:11:41.980314 27890 raft_consensus.cc:2454] T 0d9538e4316a403e805ba5d06e6585d8 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:11:41.980775 27697 leader_election.cc:302] T 0d9538e4316a403e805ba5d06e6585d8 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:41.981083 29171 raft_consensus.cc:2781] T 0d9538e4316a403e805ba5d06e6585d8 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:41.981173 29171 raft_consensus.cc:482] T 0d9538e4316a403e805ba5d06e6585d8 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:41.981289 29171 raft_consensus.cc:3037] T 0d9538e4316a403e805ba5d06e6585d8 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:41.983163 29171 raft_consensus.cc:504] T 0d9538e4316a403e805ba5d06e6585d8 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:41.983752 29171 leader_election.cc:288] T 0d9538e4316a403e805ba5d06e6585d8 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Requested vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:11:41.984308 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9538e4316a403e805ba5d06e6585d8" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:41.984299 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9538e4316a403e805ba5d06e6585d8" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:11:41.984555 28022 raft_consensus.cc:3037] T 0d9538e4316a403e805ba5d06e6585d8 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:41.984560 27890 raft_consensus.cc:3037] T 0d9538e4316a403e805ba5d06e6585d8 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:41.987532 27890 raft_consensus.cc:2454] T 0d9538e4316a403e805ba5d06e6585d8 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:11:41.987532 28022 raft_consensus.cc:2454] T 0d9538e4316a403e805ba5d06e6585d8 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:11:41.987941 27696 leader_election.cc:302] T 0d9538e4316a403e805ba5d06e6585d8 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:41.988207 29171 raft_consensus.cc:2781] T 0d9538e4316a403e805ba5d06e6585d8 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:41.988379 29171 raft_consensus.cc:686] T 0d9538e4316a403e805ba5d06e6585d8 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:11:41.988662 29171 consensus_queue.cc:237] T 0d9538e4316a403e805ba5d06e6585d8 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:41.991376 27615 catalog_manager.cc:5484] T 0d9538e4316a403e805ba5d06e6585d8 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:42.167634 29171 consensus_queue.cc:1035] T e258bb4e672f4795a3b6739f8636716e P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:42.188166 29171 consensus_queue.cc:1035] T e258bb4e672f4795a3b6739f8636716e P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:42.303462 27761 raft_consensus.cc:3037] T 20425913301a4d8abb9621cc7db0e3e7 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:42.306365 29232 consensus_queue.cc:1035] T 20425913301a4d8abb9621cc7db0e3e7 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:42.312669 29232 consensus_queue.cc:1035] T 20425913301a4d8abb9621cc7db0e3e7 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:42.368800 29285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:42.369179 29285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:42.377584 29285 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:11:42.389381 29208 consensus_queue.cc:1035] T 0d9538e4316a403e805ba5d06e6585d8 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:42.395099 29289 consensus_queue.cc:1035] T 0d9538e4316a403e805ba5d06e6585d8 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_6'
INSERT report
rows total: 2000
time total: 266.568 ms
time per row: 0.133284 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240501 08:11:43.752362 27618 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:43.761379 27870 tablet_service.cc:1467] Processing CreateTablet for tablet 728616fd583a4afea00e8fe52880e55b (DEFAULT_TABLE table=rebalance_test_table_7 [id=1ae463b433bd4ed6bb9ab1bdc10dfdf6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:43.761528 27740 tablet_service.cc:1467] Processing CreateTablet for tablet 728616fd583a4afea00e8fe52880e55b (DEFAULT_TABLE table=rebalance_test_table_7 [id=1ae463b433bd4ed6bb9ab1bdc10dfdf6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:43.761560 28131 tablet_service.cc:1467] Processing CreateTablet for tablet 728616fd583a4afea00e8fe52880e55b (DEFAULT_TABLE table=rebalance_test_table_7 [id=1ae463b433bd4ed6bb9ab1bdc10dfdf6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:43.761900 27740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 728616fd583a4afea00e8fe52880e55b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:43.761883 27870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 728616fd583a4afea00e8fe52880e55b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:43.762045 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 728616fd583a4afea00e8fe52880e55b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:43.765339 27739 tablet_service.cc:1467] Processing CreateTablet for tablet 6c04880ea87d4894bd3a63d94f4a69f4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=1ae463b433bd4ed6bb9ab1bdc10dfdf6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:43.765594 28000 tablet_service.cc:1467] Processing CreateTablet for tablet 367847f920184dd6abcdcd7564f76376 (DEFAULT_TABLE table=rebalance_test_table_7 [id=1ae463b433bd4ed6bb9ab1bdc10dfdf6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:43.765751 27739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c04880ea87d4894bd3a63d94f4a69f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:43.766016 28000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 367847f920184dd6abcdcd7564f76376. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:43.766037 27872 tablet_service.cc:1467] Processing CreateTablet for tablet 367847f920184dd6abcdcd7564f76376 (DEFAULT_TABLE table=rebalance_test_table_7 [id=1ae463b433bd4ed6bb9ab1bdc10dfdf6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:43.766438 27872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 367847f920184dd6abcdcd7564f76376. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:43.765596 28002 tablet_service.cc:1467] Processing CreateTablet for tablet 6c04880ea87d4894bd3a63d94f4a69f4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=1ae463b433bd4ed6bb9ab1bdc10dfdf6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:43.771095 28002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c04880ea87d4894bd3a63d94f4a69f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:43.775018 29351 tablet_bootstrap.cc:492] T 6c04880ea87d4894bd3a63d94f4a69f4 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:43.775251 28132 tablet_service.cc:1467] Processing CreateTablet for tablet 367847f920184dd6abcdcd7564f76376 (DEFAULT_TABLE table=rebalance_test_table_7 [id=1ae463b433bd4ed6bb9ab1bdc10dfdf6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:43.775490 29350 tablet_bootstrap.cc:492] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:43.775246 28130 tablet_service.cc:1467] Processing CreateTablet for tablet 6c04880ea87d4894bd3a63d94f4a69f4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=1ae463b433bd4ed6bb9ab1bdc10dfdf6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:43.775652 28132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 367847f920184dd6abcdcd7564f76376. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:43.775869 28130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c04880ea87d4894bd3a63d94f4a69f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:43.776590 29351 tablet_bootstrap.cc:654] T 6c04880ea87d4894bd3a63d94f4a69f4 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:43.777235 29350 tablet_bootstrap.cc:654] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:43.798473 29350 tablet_bootstrap.cc:492] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:43.798907 29350 ts_tablet_manager.cc:1367] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.009s
I20240501 08:11:43.800132 29350 raft_consensus.cc:348] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:43.800604 29350 raft_consensus.cc:374] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:43.800782 29350 raft_consensus.cc:729] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:43.800868 29355 tablet_bootstrap.cc:492] T 367847f920184dd6abcdcd7564f76376 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:43.801180 29350 consensus_queue.cc:260] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:43.802821 29355 tablet_bootstrap.cc:654] T 367847f920184dd6abcdcd7564f76376 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:43.803413 29350 ts_tablet_manager.cc:1398] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:11:43.803861 29350 tablet_bootstrap.cc:492] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:43.807350 29350 tablet_bootstrap.cc:654] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:43.812122 29354 tablet_bootstrap.cc:492] T 728616fd583a4afea00e8fe52880e55b P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:43.812217 29355 tablet_bootstrap.cc:492] T 367847f920184dd6abcdcd7564f76376 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:43.812356 29355 ts_tablet_manager.cc:1367] T 367847f920184dd6abcdcd7564f76376 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.005s
I20240501 08:11:43.812976 29350 tablet_bootstrap.cc:492] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:43.813181 29350 ts_tablet_manager.cc:1367] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20240501 08:11:43.813210 29355 raft_consensus.cc:348] T 367847f920184dd6abcdcd7564f76376 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:43.813864 29350 raft_consensus.cc:348] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:43.814056 29354 tablet_bootstrap.cc:654] T 728616fd583a4afea00e8fe52880e55b P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:43.814101 29350 raft_consensus.cc:374] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:43.814190 29350 raft_consensus.cc:729] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:43.814431 29350 consensus_queue.cc:260] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:43.814929 29350 ts_tablet_manager.cc:1398] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:11:43.815224 29355 raft_consensus.cc:374] T 367847f920184dd6abcdcd7564f76376 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:43.815356 29355 raft_consensus.cc:729] T 367847f920184dd6abcdcd7564f76376 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:43.815604 29355 consensus_queue.cc:260] T 367847f920184dd6abcdcd7564f76376 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:43.816167 29355 ts_tablet_manager.cc:1398] T 367847f920184dd6abcdcd7564f76376 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 08:11:43.816473 29355 tablet_bootstrap.cc:492] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:43.816767 29351 tablet_bootstrap.cc:492] T 6c04880ea87d4894bd3a63d94f4a69f4 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:43.816977 29351 ts_tablet_manager.cc:1367] T 6c04880ea87d4894bd3a63d94f4a69f4 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.042s user 0.014s sys 0.018s
I20240501 08:11:43.817914 29351 raft_consensus.cc:348] T 6c04880ea87d4894bd3a63d94f4a69f4 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:43.818274 29351 raft_consensus.cc:374] T 6c04880ea87d4894bd3a63d94f4a69f4 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:43.818356 29355 tablet_bootstrap.cc:654] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:43.818399 29351 raft_consensus.cc:729] T 6c04880ea87d4894bd3a63d94f4a69f4 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:43.818683 29351 consensus_queue.cc:260] T 6c04880ea87d4894bd3a63d94f4a69f4 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:43.819415 29354 tablet_bootstrap.cc:492] T 728616fd583a4afea00e8fe52880e55b P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:43.819412 29351 ts_tablet_manager.cc:1398] T 6c04880ea87d4894bd3a63d94f4a69f4 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:43.819587 29354 ts_tablet_manager.cc:1367] T 728616fd583a4afea00e8fe52880e55b P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20240501 08:11:43.819679 29351 tablet_bootstrap.cc:492] T 728616fd583a4afea00e8fe52880e55b P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:43.820354 29354 raft_consensus.cc:348] T 728616fd583a4afea00e8fe52880e55b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:43.820600 29354 raft_consensus.cc:374] T 728616fd583a4afea00e8fe52880e55b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:43.820720 29354 raft_consensus.cc:729] T 728616fd583a4afea00e8fe52880e55b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:43.820960 29354 consensus_queue.cc:260] T 728616fd583a4afea00e8fe52880e55b P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:43.821352 29351 tablet_bootstrap.cc:654] T 728616fd583a4afea00e8fe52880e55b P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:43.821507 29354 ts_tablet_manager.cc:1398] T 728616fd583a4afea00e8fe52880e55b P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:11:43.821808 29354 tablet_bootstrap.cc:492] T 6c04880ea87d4894bd3a63d94f4a69f4 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:43.826263 29354 tablet_bootstrap.cc:654] T 6c04880ea87d4894bd3a63d94f4a69f4 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:43.831378 29351 tablet_bootstrap.cc:492] T 728616fd583a4afea00e8fe52880e55b P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:43.831590 29354 tablet_bootstrap.cc:492] T 6c04880ea87d4894bd3a63d94f4a69f4 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:43.831620 29351 ts_tablet_manager.cc:1367] T 728616fd583a4afea00e8fe52880e55b P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.002s
I20240501 08:11:43.831772 29354 ts_tablet_manager.cc:1367] T 6c04880ea87d4894bd3a63d94f4a69f4 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20240501 08:11:43.832075 29355 tablet_bootstrap.cc:492] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:43.832247 29355 ts_tablet_manager.cc:1367] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.001s
I20240501 08:11:43.832285 29351 raft_consensus.cc:348] T 728616fd583a4afea00e8fe52880e55b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:43.832499 29351 raft_consensus.cc:374] T 728616fd583a4afea00e8fe52880e55b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:43.832578 29351 raft_consensus.cc:729] T 728616fd583a4afea00e8fe52880e55b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:43.832525 29354 raft_consensus.cc:348] T 6c04880ea87d4894bd3a63d94f4a69f4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:43.832754 29354 raft_consensus.cc:374] T 6c04880ea87d4894bd3a63d94f4a69f4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:43.832839 29354 raft_consensus.cc:729] T 6c04880ea87d4894bd3a63d94f4a69f4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:43.832815 29351 consensus_queue.cc:260] T 728616fd583a4afea00e8fe52880e55b P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:43.832993 29355 raft_consensus.cc:348] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:43.833221 29355 raft_consensus.cc:374] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:43.833122 29354 consensus_queue.cc:260] T 6c04880ea87d4894bd3a63d94f4a69f4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:43.833284 29355 raft_consensus.cc:729] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:43.833365 29351 ts_tablet_manager.cc:1398] T 728616fd583a4afea00e8fe52880e55b P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:11:43.833667 29355 consensus_queue.cc:260] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:43.833896 29354 ts_tablet_manager.cc:1398] T 6c04880ea87d4894bd3a63d94f4a69f4 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:43.834208 29355 ts_tablet_manager.cc:1398] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:43.836107 29354 tablet_bootstrap.cc:492] T 367847f920184dd6abcdcd7564f76376 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:43.837674 29354 tablet_bootstrap.cc:654] T 367847f920184dd6abcdcd7564f76376 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:43.840299 29354 tablet_bootstrap.cc:492] T 367847f920184dd6abcdcd7564f76376 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:43.840416 29354 ts_tablet_manager.cc:1367] T 367847f920184dd6abcdcd7564f76376 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:11:43.841110 29354 raft_consensus.cc:348] T 367847f920184dd6abcdcd7564f76376 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:43.841272 29354 raft_consensus.cc:374] T 367847f920184dd6abcdcd7564f76376 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:43.841334 29354 raft_consensus.cc:729] T 367847f920184dd6abcdcd7564f76376 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:43.841483 29354 consensus_queue.cc:260] T 367847f920184dd6abcdcd7564f76376 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:43.841871 29354 ts_tablet_manager.cc:1398] T 367847f920184dd6abcdcd7564f76376 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:11:43.844453 28797 raft_consensus.cc:482] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:43.844592 28797 raft_consensus.cc:504] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:43.845235 28797 leader_election.cc:288] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:43.845568 28152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "728616fd583a4afea00e8fe52880e55b" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:11:43.845764 28152 raft_consensus.cc:2454] T 728616fd583a4afea00e8fe52880e55b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:11:43.845748 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "728616fd583a4afea00e8fe52880e55b" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:11:43.845958 27761 raft_consensus.cc:2454] T 728616fd583a4afea00e8fe52880e55b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:11:43.846170 27825 leader_election.cc:302] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:43.846455 28797 raft_consensus.cc:2781] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:43.846542 28797 raft_consensus.cc:482] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:43.846607 28797 raft_consensus.cc:3037] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:43.848469 28797 raft_consensus.cc:504] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:43.849138 28797 leader_election.cc:288] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Requested vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:43.849514 28152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "728616fd583a4afea00e8fe52880e55b" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:11:43.849743 28152 raft_consensus.cc:3037] T 728616fd583a4afea00e8fe52880e55b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:43.849679 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "728616fd583a4afea00e8fe52880e55b" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:11:43.849884 27761 raft_consensus.cc:3037] T 728616fd583a4afea00e8fe52880e55b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:43.852638 28152 raft_consensus.cc:2454] T 728616fd583a4afea00e8fe52880e55b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:11:43.852655 27761 raft_consensus.cc:2454] T 728616fd583a4afea00e8fe52880e55b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:11:43.853039 27825 leader_election.cc:302] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:43.853345 28797 raft_consensus.cc:2781] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:43.853670 28797 raft_consensus.cc:686] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Becoming Leader. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Running, Role: LEADER
I20240501 08:11:43.853993 28797 consensus_queue.cc:237] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:43.856521 27619 catalog_manager.cc:5484] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af reported cstate change: term changed from 0 to 1, leader changed from <none> to f1525cbc82a24d13b65f136695c903af (127.21.84.130). New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:43.953366 29282 raft_consensus.cc:482] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:43.953572 29282 raft_consensus.cc:504] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:43.954319 29282 leader_election.cc:288] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:43.954818 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c04880ea87d4894bd3a63d94f4a69f4" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:11:43.954842 28152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c04880ea87d4894bd3a63d94f4a69f4" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:11:43.955067 28152 raft_consensus.cc:2454] T 6c04880ea87d4894bd3a63d94f4a69f4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 0.
I20240501 08:11:43.955085 27761 raft_consensus.cc:2454] T 6c04880ea87d4894bd3a63d94f4a69f4 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 0.
I20240501 08:11:43.955492 27955 leader_election.cc:302] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:43.955770 29282 raft_consensus.cc:2781] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:43.955873 29282 raft_consensus.cc:482] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:43.955976 29282 raft_consensus.cc:3037] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:43.957957 29282 raft_consensus.cc:504] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:43.958631 29282 leader_election.cc:288] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Requested vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:43.959050 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c04880ea87d4894bd3a63d94f4a69f4" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:11:43.959111 28152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c04880ea87d4894bd3a63d94f4a69f4" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:11:43.959326 28152 raft_consensus.cc:3037] T 6c04880ea87d4894bd3a63d94f4a69f4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:43.959285 27759 raft_consensus.cc:3037] T 6c04880ea87d4894bd3a63d94f4a69f4 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:43.961264 27759 raft_consensus.cc:2454] T 6c04880ea87d4894bd3a63d94f4a69f4 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 1.
I20240501 08:11:43.961262 28152 raft_consensus.cc:2454] T 6c04880ea87d4894bd3a63d94f4a69f4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 1.
I20240501 08:11:43.961719 27957 leader_election.cc:302] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:43.962044 29282 raft_consensus.cc:2781] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:43.962394 29282 raft_consensus.cc:686] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Becoming Leader. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Running, Role: LEADER
I20240501 08:11:43.962689 29282 consensus_queue.cc:237] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:43.965556 27619 catalog_manager.cc:5484] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to aad81047009a4ffcbc450b60030583b5 (127.21.84.131). New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:44.087167 28797 raft_consensus.cc:482] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:44.087376 28797 raft_consensus.cc:504] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:44.088066 28797 leader_election.cc:288] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:11:44.088639 28020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367847f920184dd6abcdcd7564f76376" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:44.088614 28152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367847f920184dd6abcdcd7564f76376" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:11:44.088851 28020 raft_consensus.cc:2454] T 367847f920184dd6abcdcd7564f76376 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:11:44.088866 28152 raft_consensus.cc:2454] T 367847f920184dd6abcdcd7564f76376 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:11:44.089329 27825 leader_election.cc:302] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:44.089818 28797 raft_consensus.cc:2781] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:44.089958 28797 raft_consensus.cc:482] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:44.090067 28797 raft_consensus.cc:3037] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:44.093037 28797 raft_consensus.cc:504] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:44.093783 28797 leader_election.cc:288] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Requested vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:11:44.094185 28020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367847f920184dd6abcdcd7564f76376" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:44.094383 28020 raft_consensus.cc:3037] T 367847f920184dd6abcdcd7564f76376 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:44.094491 28152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367847f920184dd6abcdcd7564f76376" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:11:44.094708 28152 raft_consensus.cc:3037] T 367847f920184dd6abcdcd7564f76376 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:44.096376 28020 raft_consensus.cc:2454] T 367847f920184dd6abcdcd7564f76376 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:11:44.096799 27827 leader_election.cc:302] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aad81047009a4ffcbc450b60030583b5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:44.097102 28797 raft_consensus.cc:2781] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:44.097260 28797 raft_consensus.cc:686] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Becoming Leader. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Running, Role: LEADER
I20240501 08:11:44.097466 28152 raft_consensus.cc:2454] T 367847f920184dd6abcdcd7564f76376 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:11:44.097584 28797 consensus_queue.cc:237] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:44.100804 27619 catalog_manager.cc:5484] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af reported cstate change: term changed from 0 to 1, leader changed from <none> to f1525cbc82a24d13b65f136695c903af (127.21.84.130). New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:44.430337 28797 consensus_queue.cc:1035] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:44.436334 28797 consensus_queue.cc:1035] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:44.507447 29373 consensus_queue.cc:1035] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:44.511803 29368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:44.512317 29368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 08:11:44.512423 28797 consensus_queue.cc:1035] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:44.521800 29282 consensus_queue.cc:1035] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:44.523677 29368 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:11:44.528257 29384 consensus_queue.cc:1035] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_7'
W20240501 08:11:44.785295 27825 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.132:41323, user_credentials={real_user=slave}} blocked reactor thread for 58155.2us
INSERT report
rows total: 2000
time total: 268.573 ms
time per row: 0.134287 ms
I20240501 08:11:44.942425 27616 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:44.951288 27740 tablet_service.cc:1467] Processing CreateTablet for tablet 44f9bd2dc83a4533afc4cff8e93164e3 (DEFAULT_TABLE table=rebalance_test_table_8 [id=ea84ae8db01148a6896438374a278971]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:44.951292 28000 tablet_service.cc:1467] Processing CreateTablet for tablet 44f9bd2dc83a4533afc4cff8e93164e3 (DEFAULT_TABLE table=rebalance_test_table_8 [id=ea84ae8db01148a6896438374a278971]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:44.951761 27740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44f9bd2dc83a4533afc4cff8e93164e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:44.951776 28000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44f9bd2dc83a4533afc4cff8e93164e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:44.951859 28131 tablet_service.cc:1467] Processing CreateTablet for tablet 3b24297ce5da4790877c5f5672a73d8c (DEFAULT_TABLE table=rebalance_test_table_8 [id=ea84ae8db01148a6896438374a278971]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:44.952237 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b24297ce5da4790877c5f5672a73d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:44.952405 27739 tablet_service.cc:1467] Processing CreateTablet for tablet 3b24297ce5da4790877c5f5672a73d8c (DEFAULT_TABLE table=rebalance_test_table_8 [id=ea84ae8db01148a6896438374a278971]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:44.952766 27739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b24297ce5da4790877c5f5672a73d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:44.954433 27872 tablet_service.cc:1467] Processing CreateTablet for tablet 44f9bd2dc83a4533afc4cff8e93164e3 (DEFAULT_TABLE table=rebalance_test_table_8 [id=ea84ae8db01148a6896438374a278971]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:44.954823 27872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44f9bd2dc83a4533afc4cff8e93164e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:44.962797 27870 tablet_service.cc:1467] Processing CreateTablet for tablet 3b24297ce5da4790877c5f5672a73d8c (DEFAULT_TABLE table=rebalance_test_table_8 [id=ea84ae8db01148a6896438374a278971]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:44.963322 27870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b24297ce5da4790877c5f5672a73d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:44.966666 29438 tablet_bootstrap.cc:492] T 44f9bd2dc83a4533afc4cff8e93164e3 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:44.967039 28002 tablet_service.cc:1467] Processing CreateTablet for tablet dfb81aba648d4099a949064b7f4442be (DEFAULT_TABLE table=rebalance_test_table_8 [id=ea84ae8db01148a6896438374a278971]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:44.967193 27741 tablet_service.cc:1467] Processing CreateTablet for tablet dfb81aba648d4099a949064b7f4442be (DEFAULT_TABLE table=rebalance_test_table_8 [id=ea84ae8db01148a6896438374a278971]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:44.967500 28002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfb81aba648d4099a949064b7f4442be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:44.967561 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfb81aba648d4099a949064b7f4442be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:44.968899 29438 tablet_bootstrap.cc:654] T 44f9bd2dc83a4533afc4cff8e93164e3 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:44.971437 28132 tablet_service.cc:1467] Processing CreateTablet for tablet dfb81aba648d4099a949064b7f4442be (DEFAULT_TABLE table=rebalance_test_table_8 [id=ea84ae8db01148a6896438374a278971]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:44.971879 28132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfb81aba648d4099a949064b7f4442be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:44.979013 29439 tablet_bootstrap.cc:492] T 44f9bd2dc83a4533afc4cff8e93164e3 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:44.981689 29439 tablet_bootstrap.cc:654] T 44f9bd2dc83a4533afc4cff8e93164e3 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:44.984236 29440 tablet_bootstrap.cc:492] T 3b24297ce5da4790877c5f5672a73d8c P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:44.986171 29440 tablet_bootstrap.cc:654] T 3b24297ce5da4790877c5f5672a73d8c P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:44.991307 29439 tablet_bootstrap.cc:492] T 44f9bd2dc83a4533afc4cff8e93164e3 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:44.991418 29437 tablet_bootstrap.cc:492] T 44f9bd2dc83a4533afc4cff8e93164e3 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:44.991539 29439 ts_tablet_manager.cc:1367] T 44f9bd2dc83a4533afc4cff8e93164e3 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20240501 08:11:44.992259 29439 raft_consensus.cc:348] T 44f9bd2dc83a4533afc4cff8e93164e3 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:44.992436 29439 raft_consensus.cc:374] T 44f9bd2dc83a4533afc4cff8e93164e3 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:44.992542 29439 raft_consensus.cc:729] T 44f9bd2dc83a4533afc4cff8e93164e3 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:44.992744 29439 consensus_queue.cc:260] T 44f9bd2dc83a4533afc4cff8e93164e3 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:44.993214 29439 ts_tablet_manager.cc:1398] T 44f9bd2dc83a4533afc4cff8e93164e3 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:11:44.993503 29439 tablet_bootstrap.cc:492] T 3b24297ce5da4790877c5f5672a73d8c P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:44.993556 29437 tablet_bootstrap.cc:654] T 44f9bd2dc83a4533afc4cff8e93164e3 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:44.995796 29439 tablet_bootstrap.cc:654] T 3b24297ce5da4790877c5f5672a73d8c P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:44.997633 29439 tablet_bootstrap.cc:492] T 3b24297ce5da4790877c5f5672a73d8c P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:44.997856 29439 ts_tablet_manager.cc:1367] T 3b24297ce5da4790877c5f5672a73d8c P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:11:44.998761 29439 raft_consensus.cc:348] T 3b24297ce5da4790877c5f5672a73d8c P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:44.999048 29439 raft_consensus.cc:374] T 3b24297ce5da4790877c5f5672a73d8c P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:44.999177 29439 raft_consensus.cc:729] T 3b24297ce5da4790877c5f5672a73d8c P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:44.999496 29439 consensus_queue.cc:260] T 3b24297ce5da4790877c5f5672a73d8c P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:45.000214 29439 ts_tablet_manager.cc:1398] T 3b24297ce5da4790877c5f5672a73d8c P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:45.000926 29440 tablet_bootstrap.cc:492] T 3b24297ce5da4790877c5f5672a73d8c P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:45.001072 29440 ts_tablet_manager.cc:1367] T 3b24297ce5da4790877c5f5672a73d8c P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.006s
I20240501 08:11:45.001202 29438 tablet_bootstrap.cc:492] T 44f9bd2dc83a4533afc4cff8e93164e3 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:45.001632 29438 ts_tablet_manager.cc:1367] T 44f9bd2dc83a4533afc4cff8e93164e3 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.035s user 0.016s sys 0.005s
I20240501 08:11:45.001847 29440 raft_consensus.cc:348] T 3b24297ce5da4790877c5f5672a73d8c P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:45.002072 29440 raft_consensus.cc:374] T 3b24297ce5da4790877c5f5672a73d8c P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:45.002179 29440 raft_consensus.cc:729] T 3b24297ce5da4790877c5f5672a73d8c P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:45.002424 29440 consensus_queue.cc:260] T 3b24297ce5da4790877c5f5672a73d8c P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:45.002503 29438 raft_consensus.cc:348] T 44f9bd2dc83a4533afc4cff8e93164e3 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:45.002802 29438 raft_consensus.cc:374] T 44f9bd2dc83a4533afc4cff8e93164e3 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:45.002918 29438 raft_consensus.cc:729] T 44f9bd2dc83a4533afc4cff8e93164e3 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:45.003013 29440 ts_tablet_manager.cc:1398] T 3b24297ce5da4790877c5f5672a73d8c P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:11:45.003175 29438 consensus_queue.cc:260] T 44f9bd2dc83a4533afc4cff8e93164e3 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:45.003352 29440 tablet_bootstrap.cc:492] T dfb81aba648d4099a949064b7f4442be P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:45.003721 29438 ts_tablet_manager.cc:1398] T 44f9bd2dc83a4533afc4cff8e93164e3 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:11:45.003952 29438 tablet_bootstrap.cc:492] T dfb81aba648d4099a949064b7f4442be P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:45.005000 29440 tablet_bootstrap.cc:654] T dfb81aba648d4099a949064b7f4442be P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:45.005323 29438 tablet_bootstrap.cc:654] T dfb81aba648d4099a949064b7f4442be P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:45.007189 29437 tablet_bootstrap.cc:492] T 44f9bd2dc83a4533afc4cff8e93164e3 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:45.007347 29437 ts_tablet_manager.cc:1367] T 44f9bd2dc83a4533afc4cff8e93164e3 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.004s
I20240501 08:11:45.008258 29437 raft_consensus.cc:348] T 44f9bd2dc83a4533afc4cff8e93164e3 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:45.008476 29437 raft_consensus.cc:374] T 44f9bd2dc83a4533afc4cff8e93164e3 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:45.008620 29437 raft_consensus.cc:729] T 44f9bd2dc83a4533afc4cff8e93164e3 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:45.008905 29437 consensus_queue.cc:260] T 44f9bd2dc83a4533afc4cff8e93164e3 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:45.009667 29437 ts_tablet_manager.cc:1398] T 44f9bd2dc83a4533afc4cff8e93164e3 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:11:45.011749 28797 raft_consensus.cc:482] T 44f9bd2dc83a4533afc4cff8e93164e3 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:45.011996 28797 raft_consensus.cc:504] T 44f9bd2dc83a4533afc4cff8e93164e3 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:45.012907 28797 leader_election.cc:288] T 44f9bd2dc83a4533afc4cff8e93164e3 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:45.013762 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f9bd2dc83a4533afc4cff8e93164e3" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:11:45.014078 27759 raft_consensus.cc:2454] T 44f9bd2dc83a4533afc4cff8e93164e3 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:11:45.014600 27827 leader_election.cc:302] T 44f9bd2dc83a4533afc4cff8e93164e3 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:45.014999 28797 raft_consensus.cc:2781] T 44f9bd2dc83a4533afc4cff8e93164e3 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:45.015113 28797 raft_consensus.cc:482] T 44f9bd2dc83a4533afc4cff8e93164e3 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:45.015259 28797 raft_consensus.cc:3037] T 44f9bd2dc83a4533afc4cff8e93164e3 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:45.019075 28797 raft_consensus.cc:504] T 44f9bd2dc83a4533afc4cff8e93164e3 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:45.020076 29438 tablet_bootstrap.cc:492] T dfb81aba648d4099a949064b7f4442be P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:45.020354 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f9bd2dc83a4533afc4cff8e93164e3" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:45.020576 28021 raft_consensus.cc:3037] T 44f9bd2dc83a4533afc4cff8e93164e3 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:45.020814 29440 tablet_bootstrap.cc:492] T dfb81aba648d4099a949064b7f4442be P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:45.020881 29437 tablet_bootstrap.cc:492] T dfb81aba648d4099a949064b7f4442be P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:45.020987 29440 ts_tablet_manager.cc:1367] T dfb81aba648d4099a949064b7f4442be P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.010s
I20240501 08:11:45.021811 29440 raft_consensus.cc:348] T dfb81aba648d4099a949064b7f4442be P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:45.022053 29440 raft_consensus.cc:374] T dfb81aba648d4099a949064b7f4442be P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:45.022130 29440 raft_consensus.cc:729] T dfb81aba648d4099a949064b7f4442be P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:45.022410 29438 ts_tablet_manager.cc:1367] T dfb81aba648d4099a949064b7f4442be P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.001s
I20240501 08:11:45.023165 29438 raft_consensus.cc:348] T dfb81aba648d4099a949064b7f4442be P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:45.023345 28021 raft_consensus.cc:2454] T 44f9bd2dc83a4533afc4cff8e93164e3 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:11:45.023381 29438 raft_consensus.cc:374] T dfb81aba648d4099a949064b7f4442be P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:45.023444 29438 raft_consensus.cc:729] T dfb81aba648d4099a949064b7f4442be P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:45.023548 29437 tablet_bootstrap.cc:654] T dfb81aba648d4099a949064b7f4442be P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:45.023689 29438 consensus_queue.cc:260] T dfb81aba648d4099a949064b7f4442be P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:45.024130 29440 consensus_queue.cc:260] T dfb81aba648d4099a949064b7f4442be P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:45.024408 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f9bd2dc83a4533afc4cff8e93164e3" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:45.024807 29438 ts_tablet_manager.cc:1398] T dfb81aba648d4099a949064b7f4442be P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:11:45.025102 29438 tablet_bootstrap.cc:492] T 3b24297ce5da4790877c5f5672a73d8c P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:45.027091 29440 ts_tablet_manager.cc:1398] T dfb81aba648d4099a949064b7f4442be P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:11:45.027421 29438 tablet_bootstrap.cc:654] T 3b24297ce5da4790877c5f5672a73d8c P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:45.033739 27827 leader_election.cc:302] T 44f9bd2dc83a4533afc4cff8e93164e3 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aad81047009a4ffcbc450b60030583b5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:45.034216 29363 raft_consensus.cc:2781] T 44f9bd2dc83a4533afc4cff8e93164e3 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:45.034425 29363 raft_consensus.cc:686] T 44f9bd2dc83a4533afc4cff8e93164e3 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Becoming Leader. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Running, Role: LEADER
I20240501 08:11:45.034801 29363 consensus_queue.cc:237] T 44f9bd2dc83a4533afc4cff8e93164e3 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:45.052840 27619 catalog_manager.cc:5484] T 44f9bd2dc83a4533afc4cff8e93164e3 P f1525cbc82a24d13b65f136695c903af reported cstate change: term changed from 0 to 1, leader changed from <none> to f1525cbc82a24d13b65f136695c903af (127.21.84.130). New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:45.059587 29437 tablet_bootstrap.cc:492] T dfb81aba648d4099a949064b7f4442be P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:45.059862 29437 ts_tablet_manager.cc:1367] T dfb81aba648d4099a949064b7f4442be P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.039s user 0.016s sys 0.016s
I20240501 08:11:45.060460 29438 tablet_bootstrap.cc:492] T 3b24297ce5da4790877c5f5672a73d8c P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:45.060678 29438 ts_tablet_manager.cc:1367] T 3b24297ce5da4790877c5f5672a73d8c P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.036s user 0.011s sys 0.023s
I20240501 08:11:45.060752 29437 raft_consensus.cc:348] T dfb81aba648d4099a949064b7f4442be P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:45.061064 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f9bd2dc83a4533afc4cff8e93164e3" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:11:45.061174 29437 raft_consensus.cc:374] T dfb81aba648d4099a949064b7f4442be P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:45.061146 28797 leader_election.cc:288] T 44f9bd2dc83a4533afc4cff8e93164e3 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Requested vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:45.061247 29437 raft_consensus.cc:729] T dfb81aba648d4099a949064b7f4442be P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:45.061292 27759 raft_consensus.cc:3037] T 44f9bd2dc83a4533afc4cff8e93164e3 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:45.061535 29437 consensus_queue.cc:260] T dfb81aba648d4099a949064b7f4442be P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:45.062291 29437 ts_tablet_manager.cc:1398] T dfb81aba648d4099a949064b7f4442be P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:11:45.063235 29438 raft_consensus.cc:348] T 3b24297ce5da4790877c5f5672a73d8c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:45.063434 29438 raft_consensus.cc:374] T 3b24297ce5da4790877c5f5672a73d8c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:45.063536 29438 raft_consensus.cc:729] T 3b24297ce5da4790877c5f5672a73d8c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:45.063794 29438 consensus_queue.cc:260] T 3b24297ce5da4790877c5f5672a73d8c P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:45.064142 27759 raft_consensus.cc:2454] T 44f9bd2dc83a4533afc4cff8e93164e3 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:11:45.065213 29438 ts_tablet_manager.cc:1398] T 3b24297ce5da4790877c5f5672a73d8c P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 08:11:45.080989 29344 raft_consensus.cc:482] T dfb81aba648d4099a949064b7f4442be P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:45.081168 29344 raft_consensus.cc:504] T dfb81aba648d4099a949064b7f4442be P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:45.081915 29344 leader_election.cc:288] T dfb81aba648d4099a949064b7f4442be P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:45.082628 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfb81aba648d4099a949064b7f4442be" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:45.082870 28021 raft_consensus.cc:2454] T dfb81aba648d4099a949064b7f4442be P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:45.083292 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfb81aba648d4099a949064b7f4442be" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:11:45.083340 28087 leader_election.cc:302] T dfb81aba648d4099a949064b7f4442be P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:45.083515 27759 raft_consensus.cc:2454] T dfb81aba648d4099a949064b7f4442be P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:45.083618 29344 raft_consensus.cc:2781] T dfb81aba648d4099a949064b7f4442be P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:45.083719 29344 raft_consensus.cc:482] T dfb81aba648d4099a949064b7f4442be P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:45.083806 29344 raft_consensus.cc:3037] T dfb81aba648d4099a949064b7f4442be P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:45.086701 29344 raft_consensus.cc:504] T dfb81aba648d4099a949064b7f4442be P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:45.087582 29344 leader_election.cc:288] T dfb81aba648d4099a949064b7f4442be P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:45.087626 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfb81aba648d4099a949064b7f4442be" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:45.087863 28021 raft_consensus.cc:3037] T dfb81aba648d4099a949064b7f4442be P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:45.088208 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfb81aba648d4099a949064b7f4442be" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:11:45.088421 27759 raft_consensus.cc:3037] T dfb81aba648d4099a949064b7f4442be P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:45.089807 28021 raft_consensus.cc:2454] T dfb81aba648d4099a949064b7f4442be P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:45.090317 28087 leader_election.cc:302] T dfb81aba648d4099a949064b7f4442be P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:45.090605 29453 raft_consensus.cc:2781] T dfb81aba648d4099a949064b7f4442be P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:45.090844 29453 raft_consensus.cc:686] T dfb81aba648d4099a949064b7f4442be P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:11:45.091189 29453 consensus_queue.cc:237] T dfb81aba648d4099a949064b7f4442be P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:45.091238 27759 raft_consensus.cc:2454] T dfb81aba648d4099a949064b7f4442be P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:45.094521 27616 catalog_manager.cc:5484] T dfb81aba648d4099a949064b7f4442be P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:45.102073 29453 raft_consensus.cc:482] T 3b24297ce5da4790877c5f5672a73d8c P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:45.102267 29453 raft_consensus.cc:504] T 3b24297ce5da4790877c5f5672a73d8c P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:45.102993 29453 leader_election.cc:288] T 3b24297ce5da4790877c5f5672a73d8c P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:45.103509 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b24297ce5da4790877c5f5672a73d8c" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:11:45.103547 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b24297ce5da4790877c5f5672a73d8c" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:11:45.103708 27890 raft_consensus.cc:2454] T 3b24297ce5da4790877c5f5672a73d8c P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:45.103847 27759 raft_consensus.cc:2454] T 3b24297ce5da4790877c5f5672a73d8c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:45.104266 28088 leader_election.cc:302] T 3b24297ce5da4790877c5f5672a73d8c P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:45.104589 29453 raft_consensus.cc:2781] T 3b24297ce5da4790877c5f5672a73d8c P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:45.104700 29453 raft_consensus.cc:482] T 3b24297ce5da4790877c5f5672a73d8c P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:45.104815 29453 raft_consensus.cc:3037] T 3b24297ce5da4790877c5f5672a73d8c P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:45.107614 29453 raft_consensus.cc:504] T 3b24297ce5da4790877c5f5672a73d8c P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:45.108302 29453 leader_election.cc:288] T 3b24297ce5da4790877c5f5672a73d8c P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:45.108826 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b24297ce5da4790877c5f5672a73d8c" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:11:45.108808 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b24297ce5da4790877c5f5672a73d8c" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:11:45.109016 27759 raft_consensus.cc:3037] T 3b24297ce5da4790877c5f5672a73d8c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:45.109023 27890 raft_consensus.cc:3037] T 3b24297ce5da4790877c5f5672a73d8c P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:45.111044 27759 raft_consensus.cc:2454] T 3b24297ce5da4790877c5f5672a73d8c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:45.111470 28087 leader_election.cc:302] T 3b24297ce5da4790877c5f5672a73d8c P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:11:45.111758 29453 raft_consensus.cc:2781] T 3b24297ce5da4790877c5f5672a73d8c P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:45.111912 29453 raft_consensus.cc:686] T 3b24297ce5da4790877c5f5672a73d8c P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:11:45.111909 27890 raft_consensus.cc:2454] T 3b24297ce5da4790877c5f5672a73d8c P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:45.112186 29453 consensus_queue.cc:237] T 3b24297ce5da4790877c5f5672a73d8c P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:45.115458 27616 catalog_manager.cc:5484] T 3b24297ce5da4790877c5f5672a73d8c P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:45.432380 28797 consensus_queue.cc:1035] T 44f9bd2dc83a4533afc4cff8e93164e3 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:45.446610 28797 consensus_queue.cc:1035] T 44f9bd2dc83a4533afc4cff8e93164e3 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:45.491690 29456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:45.492031 29456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:45.499845 29456 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240501 08:11:45.624347 29453 consensus_queue.cc:1035] T dfb81aba648d4099a949064b7f4442be P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:45.661715 29495 consensus_queue.cc:1035] T dfb81aba648d4099a949064b7f4442be P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:45.664189 29453 consensus_queue.cc:1035] T 3b24297ce5da4790877c5f5672a73d8c P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:45.681288 27759 raft_consensus.cc:1261] T 3b24297ce5da4790877c5f5672a73d8c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:11:45.682806 29453 consensus_queue.cc:1035] T 3b24297ce5da4790877c5f5672a73d8c P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 272.179 ms
time per row: 0.13609 ms
I20240501 08:11:45.930296 27612 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:45.940538 28000 tablet_service.cc:1467] Processing CreateTablet for tablet a2981c0863bc41729605f8b076b6a6cb (DEFAULT_TABLE table=rebalance_test_table_9 [id=e6efea2767a741beae5c93339e7712ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:45.940635 27870 tablet_service.cc:1467] Processing CreateTablet for tablet a2981c0863bc41729605f8b076b6a6cb (DEFAULT_TABLE table=rebalance_test_table_9 [id=e6efea2767a741beae5c93339e7712ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:45.941058 28000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2981c0863bc41729605f8b076b6a6cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:45.941138 27870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2981c0863bc41729605f8b076b6a6cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:45.941639 27871 tablet_service.cc:1467] Processing CreateTablet for tablet 802538381de843ab8d326014a0b4ec15 (DEFAULT_TABLE table=rebalance_test_table_9 [id=e6efea2767a741beae5c93339e7712ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:45.941709 27739 tablet_service.cc:1467] Processing CreateTablet for tablet 802538381de843ab8d326014a0b4ec15 (DEFAULT_TABLE table=rebalance_test_table_9 [id=e6efea2767a741beae5c93339e7712ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:45.942044 27871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 802538381de843ab8d326014a0b4ec15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:45.942157 27739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 802538381de843ab8d326014a0b4ec15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:45.942302 27872 tablet_service.cc:1467] Processing CreateTablet for tablet a0f992f08fb841a5b18b14f5ad0b9c58 (DEFAULT_TABLE table=rebalance_test_table_9 [id=e6efea2767a741beae5c93339e7712ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:45.942670 27872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0f992f08fb841a5b18b14f5ad0b9c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:45.946377 28132 tablet_service.cc:1467] Processing CreateTablet for tablet a0f992f08fb841a5b18b14f5ad0b9c58 (DEFAULT_TABLE table=rebalance_test_table_9 [id=e6efea2767a741beae5c93339e7712ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:45.946815 28132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0f992f08fb841a5b18b14f5ad0b9c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:45.948683 28131 tablet_service.cc:1467] Processing CreateTablet for tablet a2981c0863bc41729605f8b076b6a6cb (DEFAULT_TABLE table=rebalance_test_table_9 [id=e6efea2767a741beae5c93339e7712ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:45.949086 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2981c0863bc41729605f8b076b6a6cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:45.951164 29524 tablet_bootstrap.cc:492] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:45.952980 29524 tablet_bootstrap.cc:654] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:45.953426 29526 tablet_bootstrap.cc:492] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:45.954830 28002 tablet_service.cc:1467] Processing CreateTablet for tablet a0f992f08fb841a5b18b14f5ad0b9c58 (DEFAULT_TABLE table=rebalance_test_table_9 [id=e6efea2767a741beae5c93339e7712ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:45.955279 28002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0f992f08fb841a5b18b14f5ad0b9c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:45.955440 29526 tablet_bootstrap.cc:654] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:45.960752 28001 tablet_service.cc:1467] Processing CreateTablet for tablet 802538381de843ab8d326014a0b4ec15 (DEFAULT_TABLE table=rebalance_test_table_9 [id=e6efea2767a741beae5c93339e7712ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:45.961323 28001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 802538381de843ab8d326014a0b4ec15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:45.974756 29525 tablet_bootstrap.cc:492] T a0f992f08fb841a5b18b14f5ad0b9c58 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:45.976250 29529 tablet_bootstrap.cc:492] T 802538381de843ab8d326014a0b4ec15 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:45.978782 29525 tablet_bootstrap.cc:654] T a0f992f08fb841a5b18b14f5ad0b9c58 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:45.979430 29529 tablet_bootstrap.cc:654] T 802538381de843ab8d326014a0b4ec15 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:45.981442 29526 tablet_bootstrap.cc:492] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:45.981582 29524 tablet_bootstrap.cc:492] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:45.981596 29526 ts_tablet_manager.cc:1367] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.028s user 0.017s sys 0.008s
I20240501 08:11:45.981741 29524 ts_tablet_manager.cc:1367] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.016s
I20240501 08:11:45.982455 29526 raft_consensus.cc:348] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:45.982563 29524 raft_consensus.cc:348] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:45.982705 29526 raft_consensus.cc:374] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:45.982771 29524 raft_consensus.cc:374] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:45.982818 29526 raft_consensus.cc:729] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:45.982875 29524 raft_consensus.cc:729] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:45.983067 29526 consensus_queue.cc:260] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:45.983348 29529 tablet_bootstrap.cc:492] T 802538381de843ab8d326014a0b4ec15 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:45.983487 29529 ts_tablet_manager.cc:1367] T 802538381de843ab8d326014a0b4ec15 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 08:11:45.983659 29526 ts_tablet_manager.cc:1398] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:45.983959 29526 tablet_bootstrap.cc:492] T a2981c0863bc41729605f8b076b6a6cb P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:45.984268 29529 raft_consensus.cc:348] T 802538381de843ab8d326014a0b4ec15 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:45.984483 29529 raft_consensus.cc:374] T 802538381de843ab8d326014a0b4ec15 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:45.984436 29525 tablet_bootstrap.cc:492] T a0f992f08fb841a5b18b14f5ad0b9c58 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:45.984594 29529 raft_consensus.cc:729] T 802538381de843ab8d326014a0b4ec15 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:45.984808 29525 ts_tablet_manager.cc:1367] T a0f992f08fb841a5b18b14f5ad0b9c58 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240501 08:11:45.984810 29524 consensus_queue.cc:260] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:45.984839 29529 consensus_queue.cc:260] T 802538381de843ab8d326014a0b4ec15 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:45.985417 29529 ts_tablet_manager.cc:1398] T 802538381de843ab8d326014a0b4ec15 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:11:45.985812 29529 tablet_bootstrap.cc:492] T a2981c0863bc41729605f8b076b6a6cb P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:45.985771 29525 raft_consensus.cc:348] T a0f992f08fb841a5b18b14f5ad0b9c58 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:45.986188 29525 raft_consensus.cc:374] T a0f992f08fb841a5b18b14f5ad0b9c58 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:45.986701 29524 ts_tablet_manager.cc:1398] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 08:11:45.987469 29529 tablet_bootstrap.cc:654] T a2981c0863bc41729605f8b076b6a6cb P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:45.987618 29525 raft_consensus.cc:729] T a0f992f08fb841a5b18b14f5ad0b9c58 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:45.987962 29525 consensus_queue.cc:260] T a0f992f08fb841a5b18b14f5ad0b9c58 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:45.988593 29525 ts_tablet_manager.cc:1398] T a0f992f08fb841a5b18b14f5ad0b9c58 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:11:45.988991 29525 tablet_bootstrap.cc:492] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:45.993120 29525 tablet_bootstrap.cc:654] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:45.996706 29345 raft_consensus.cc:482] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:45.996949 29345 raft_consensus.cc:504] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:45.997864 29345 leader_election.cc:288] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:11:46.000550 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "802538381de843ab8d326014a0b4ec15" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
W20240501 08:11:46.001507 27697 leader_election.cc:341] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:11:46.004683 29526 tablet_bootstrap.cc:654] T a2981c0863bc41729605f8b076b6a6cb P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:46.013679 29526 tablet_bootstrap.cc:492] T a2981c0863bc41729605f8b076b6a6cb P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:46.013959 29526 ts_tablet_manager.cc:1367] T a2981c0863bc41729605f8b076b6a6cb P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.008s
I20240501 08:11:46.015111 29526 raft_consensus.cc:348] T a2981c0863bc41729605f8b076b6a6cb P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:46.015444 29526 raft_consensus.cc:374] T a2981c0863bc41729605f8b076b6a6cb P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:46.015542 29526 raft_consensus.cc:729] T a2981c0863bc41729605f8b076b6a6cb P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:46.015996 29526 consensus_queue.cc:260] T a2981c0863bc41729605f8b076b6a6cb P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:46.020751 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "802538381de843ab8d326014a0b4ec15" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:46.021116 28022 raft_consensus.cc:2454] T 802538381de843ab8d326014a0b4ec15 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:11:46.021283 29526 ts_tablet_manager.cc:1398] T a2981c0863bc41729605f8b076b6a6cb P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240501 08:11:46.022078 27696 leader_election.cc:302] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, aad81047009a4ffcbc450b60030583b5; no voters: f1525cbc82a24d13b65f136695c903af
I20240501 08:11:46.022953 29345 raft_consensus.cc:2781] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:46.023094 29345 raft_consensus.cc:482] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:46.023207 29345 raft_consensus.cc:3037] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:46.027288 29345 raft_consensus.cc:504] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:46.028014 29345 leader_election.cc:288] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Requested vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:11:46.028470 29529 tablet_bootstrap.cc:492] T a2981c0863bc41729605f8b076b6a6cb P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:46.028498 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "802538381de843ab8d326014a0b4ec15" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:11:46.028774 29529 ts_tablet_manager.cc:1367] T a2981c0863bc41729605f8b076b6a6cb P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.043s user 0.019s sys 0.022s
I20240501 08:11:46.028903 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "802538381de843ab8d326014a0b4ec15" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
W20240501 08:11:46.029064 27697 leader_election.cc:341] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:11:46.029127 28022 raft_consensus.cc:3037] T 802538381de843ab8d326014a0b4ec15 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:46.029779 29529 raft_consensus.cc:348] T a2981c0863bc41729605f8b076b6a6cb P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:46.030225 29529 raft_consensus.cc:374] T a2981c0863bc41729605f8b076b6a6cb P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:46.030357 29529 raft_consensus.cc:729] T a2981c0863bc41729605f8b076b6a6cb P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:46.030616 29529 consensus_queue.cc:260] T a2981c0863bc41729605f8b076b6a6cb P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:46.031078 29529 ts_tablet_manager.cc:1398] T a2981c0863bc41729605f8b076b6a6cb P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:11:46.031383 29529 tablet_bootstrap.cc:492] T a0f992f08fb841a5b18b14f5ad0b9c58 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:46.032120 28022 raft_consensus.cc:2454] T 802538381de843ab8d326014a0b4ec15 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:11:46.032816 27696 leader_election.cc:302] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, aad81047009a4ffcbc450b60030583b5; no voters: f1525cbc82a24d13b65f136695c903af
I20240501 08:11:46.033003 29529 tablet_bootstrap.cc:654] T a0f992f08fb841a5b18b14f5ad0b9c58 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:46.033210 29345 raft_consensus.cc:2781] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:46.033404 29345 raft_consensus.cc:686] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:11:46.033733 29345 consensus_queue.cc:237] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:46.047143 29525 tablet_bootstrap.cc:492] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:46.047356 29525 ts_tablet_manager.cc:1367] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.058s user 0.017s sys 0.016s
I20240501 08:11:46.048207 29525 raft_consensus.cc:348] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:46.048422 29525 raft_consensus.cc:374] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:46.048503 29525 raft_consensus.cc:729] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:46.048744 29525 consensus_queue.cc:260] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:46.049412 29525 ts_tablet_manager.cc:1398] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:11:46.049746 29525 tablet_bootstrap.cc:492] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:46.057015 29491 raft_consensus.cc:482] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:46.057184 29491 raft_consensus.cc:504] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:46.057866 29491 leader_election.cc:288] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:11:46.058605 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0f992f08fb841a5b18b14f5ad0b9c58" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:46.059211 29529 tablet_bootstrap.cc:492] T a0f992f08fb841a5b18b14f5ad0b9c58 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:46.059401 29529 ts_tablet_manager.cc:1367] T a0f992f08fb841a5b18b14f5ad0b9c58 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.004s
I20240501 08:11:46.059496 29525 tablet_bootstrap.cc:654] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
W20240501 08:11:46.059587 28087 leader_election.cc:341] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:11:46.060460 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0f992f08fb841a5b18b14f5ad0b9c58" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:11:46.060410 29529 raft_consensus.cc:348] T a0f992f08fb841a5b18b14f5ad0b9c58 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:46.060678 29529 raft_consensus.cc:374] T a0f992f08fb841a5b18b14f5ad0b9c58 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:46.060719 27890 raft_consensus.cc:2454] T a0f992f08fb841a5b18b14f5ad0b9c58 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:46.060765 29529 raft_consensus.cc:729] T a0f992f08fb841a5b18b14f5ad0b9c58 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:46.060998 29529 consensus_queue.cc:260] T a0f992f08fb841a5b18b14f5ad0b9c58 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:46.061362 28088 leader_election.cc:302] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters: aad81047009a4ffcbc450b60030583b5
I20240501 08:11:46.061559 29529 ts_tablet_manager.cc:1398] T a0f992f08fb841a5b18b14f5ad0b9c58 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 08:11:46.063198 29491 raft_consensus.cc:2781] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:46.063341 29491 raft_consensus.cc:482] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:46.063416 29491 raft_consensus.cc:3037] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:46.066545 29491 raft_consensus.cc:504] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:46.067723 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0f992f08fb841a5b18b14f5ad0b9c58" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:11:46.067922 27890 raft_consensus.cc:3037] T a0f992f08fb841a5b18b14f5ad0b9c58 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:46.070863 29491 leader_election.cc:288] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:11:46.071128 27612 catalog_manager.cc:5484] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:46.071430 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0f992f08fb841a5b18b14f5ad0b9c58" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:46.071681 28022 raft_consensus.cc:3037] T a0f992f08fb841a5b18b14f5ad0b9c58 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:46.072113 29525 tablet_bootstrap.cc:492] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:46.072280 29525 ts_tablet_manager.cc:1367] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.000s
I20240501 08:11:46.073472 29525 raft_consensus.cc:348] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:46.073638 29525 raft_consensus.cc:374] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:46.073732 29525 raft_consensus.cc:729] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:46.074003 29525 consensus_queue.cc:260] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:46.074476 28022 raft_consensus.cc:2454] T a0f992f08fb841a5b18b14f5ad0b9c58 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:46.074640 29525 ts_tablet_manager.cc:1398] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240501 08:11:46.075158 28087 leader_election.cc:302] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:46.076004 29491 raft_consensus.cc:2781] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:46.076220 29491 raft_consensus.cc:686] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:11:46.076547 27890 raft_consensus.cc:2454] T a0f992f08fb841a5b18b14f5ad0b9c58 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:46.076565 29491 consensus_queue.cc:237] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:46.080900 27617 catalog_manager.cc:5484] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:46.343097 29363 raft_consensus.cc:482] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:46.343312 29363 raft_consensus.cc:504] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:46.343959 29363 leader_election.cc:288] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:11:46.344580 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2981c0863bc41729605f8b076b6a6cb" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:46.344576 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2981c0863bc41729605f8b076b6a6cb" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:11:46.344790 28022 raft_consensus.cc:2454] T a2981c0863bc41729605f8b076b6a6cb P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:11:46.344833 28151 raft_consensus.cc:2454] T a2981c0863bc41729605f8b076b6a6cb P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:11:46.345296 27827 leader_election.cc:302] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aad81047009a4ffcbc450b60030583b5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:46.345613 29363 raft_consensus.cc:2781] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:46.345739 29363 raft_consensus.cc:482] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:46.345858 29363 raft_consensus.cc:3037] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:46.347885 29363 raft_consensus.cc:504] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:46.348475 29363 leader_election.cc:288] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Requested vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:11:46.348973 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2981c0863bc41729605f8b076b6a6cb" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:11:46.349009 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2981c0863bc41729605f8b076b6a6cb" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:46.349208 28022 raft_consensus.cc:3037] T a2981c0863bc41729605f8b076b6a6cb P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:46.349218 28151 raft_consensus.cc:3037] T a2981c0863bc41729605f8b076b6a6cb P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:46.352049 28022 raft_consensus.cc:2454] T a2981c0863bc41729605f8b076b6a6cb P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:11:46.352049 28151 raft_consensus.cc:2454] T a2981c0863bc41729605f8b076b6a6cb P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:11:46.352514 27825 leader_election.cc:302] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:46.352792 29363 raft_consensus.cc:2781] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:46.352960 29363 raft_consensus.cc:686] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Becoming Leader. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Running, Role: LEADER
I20240501 08:11:46.353188 29363 consensus_queue.cc:237] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:46.357177 27612 catalog_manager.cc:5484] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af reported cstate change: term changed from 0 to 1, leader changed from <none> to f1525cbc82a24d13b65f136695c903af (127.21.84.130). New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:46.415063 29363 raft_consensus.cc:482] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:46.415258 29363 raft_consensus.cc:504] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:46.415863 29363 leader_election.cc:288] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:11:46.416388 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "802538381de843ab8d326014a0b4ec15" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:11:46.416393 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "802538381de843ab8d326014a0b4ec15" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:46.416642 28022 raft_consensus.cc:2379] T 802538381de843ab8d326014a0b4ec15 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f1525cbc82a24d13b65f136695c903af in current term 1: Already voted for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in this term.
I20240501 08:11:46.417047 27827 leader_election.cc:302] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1525cbc82a24d13b65f136695c903af; no voters: 6331f77dae9d452184e8cd0cbf5f1b43, aad81047009a4ffcbc450b60030583b5
I20240501 08:11:46.417330 29363 raft_consensus.cc:3037] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:46.419230 29363 raft_consensus.cc:2726] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:11:46.514565 29345 consensus_queue.cc:1035] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:46.521318 29345 consensus_queue.cc:1035] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:46.622607 29495 consensus_queue.cc:1035] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:46.648243 29551 consensus_queue.cc:1035] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240501 08:11:46.748844 29363 consensus_queue.cc:1035] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:46.753649 29363 consensus_queue.cc:1035] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:46.794045 29542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:46.794409 29542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:46.802762 29542 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
INSERT report
rows total: 2000
time total: 288.371 ms
time per row: 0.144186 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 08:11:48.211470 27617 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:48.221096 27872 tablet_service.cc:1467] Processing CreateTablet for tablet f8b10e23797c42c9b3ce5d45a1487f95 (DEFAULT_TABLE table=rebalance_test_table_10 [id=f32998915447419ebb7965fabf78af27]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:48.221583 27872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8b10e23797c42c9b3ce5d45a1487f95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:48.222821 28131 tablet_service.cc:1467] Processing CreateTablet for tablet 80dac7e5df9248e1bef33b75c3878477 (DEFAULT_TABLE table=rebalance_test_table_10 [id=f32998915447419ebb7965fabf78af27]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:48.222826 27870 tablet_service.cc:1467] Processing CreateTablet for tablet adf805b1ae304ebea98cfb2bf3e4eca1 (DEFAULT_TABLE table=rebalance_test_table_10 [id=f32998915447419ebb7965fabf78af27]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:48.222857 28001 tablet_service.cc:1467] Processing CreateTablet for tablet 80dac7e5df9248e1bef33b75c3878477 (DEFAULT_TABLE table=rebalance_test_table_10 [id=f32998915447419ebb7965fabf78af27]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:48.223251 27870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adf805b1ae304ebea98cfb2bf3e4eca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:48.223249 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80dac7e5df9248e1bef33b75c3878477. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:48.223281 28001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80dac7e5df9248e1bef33b75c3878477. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:48.228462 28132 tablet_service.cc:1467] Processing CreateTablet for tablet f8b10e23797c42c9b3ce5d45a1487f95 (DEFAULT_TABLE table=rebalance_test_table_10 [id=f32998915447419ebb7965fabf78af27]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:48.228933 28132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8b10e23797c42c9b3ce5d45a1487f95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:48.230494 28130 tablet_service.cc:1467] Processing CreateTablet for tablet adf805b1ae304ebea98cfb2bf3e4eca1 (DEFAULT_TABLE table=rebalance_test_table_10 [id=f32998915447419ebb7965fabf78af27]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:48.231040 28130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adf805b1ae304ebea98cfb2bf3e4eca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:48.234014 29612 tablet_bootstrap.cc:492] T f8b10e23797c42c9b3ce5d45a1487f95 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:48.235276 27739 tablet_service.cc:1467] Processing CreateTablet for tablet f8b10e23797c42c9b3ce5d45a1487f95 (DEFAULT_TABLE table=rebalance_test_table_10 [id=f32998915447419ebb7965fabf78af27]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:48.235461 27740 tablet_service.cc:1467] Processing CreateTablet for tablet adf805b1ae304ebea98cfb2bf3e4eca1 (DEFAULT_TABLE table=rebalance_test_table_10 [id=f32998915447419ebb7965fabf78af27]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:48.235725 27739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8b10e23797c42c9b3ce5d45a1487f95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:48.237977 27740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adf805b1ae304ebea98cfb2bf3e4eca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:48.238310 27741 tablet_service.cc:1467] Processing CreateTablet for tablet 80dac7e5df9248e1bef33b75c3878477 (DEFAULT_TABLE table=rebalance_test_table_10 [id=f32998915447419ebb7965fabf78af27]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:48.238688 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80dac7e5df9248e1bef33b75c3878477. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:48.240689 29614 tablet_bootstrap.cc:492] T 80dac7e5df9248e1bef33b75c3878477 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:48.243294 29614 tablet_bootstrap.cc:654] T 80dac7e5df9248e1bef33b75c3878477 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:48.245317 29614 tablet_bootstrap.cc:492] T 80dac7e5df9248e1bef33b75c3878477 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:48.245551 29614 ts_tablet_manager.cc:1367] T 80dac7e5df9248e1bef33b75c3878477 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 08:11:48.246475 29614 raft_consensus.cc:348] T 80dac7e5df9248e1bef33b75c3878477 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:48.246779 29614 raft_consensus.cc:374] T 80dac7e5df9248e1bef33b75c3878477 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:48.246912 29616 tablet_bootstrap.cc:492] T f8b10e23797c42c9b3ce5d45a1487f95 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:48.246933 29614 raft_consensus.cc:729] T 80dac7e5df9248e1bef33b75c3878477 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:48.247004 29612 tablet_bootstrap.cc:654] T f8b10e23797c42c9b3ce5d45a1487f95 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:48.247241 29614 consensus_queue.cc:260] T 80dac7e5df9248e1bef33b75c3878477 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:48.249270 29616 tablet_bootstrap.cc:654] T f8b10e23797c42c9b3ce5d45a1487f95 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:48.249295 29614 ts_tablet_manager.cc:1398] T 80dac7e5df9248e1bef33b75c3878477 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:11:48.249748 29614 tablet_bootstrap.cc:492] T adf805b1ae304ebea98cfb2bf3e4eca1 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:48.251591 29613 tablet_bootstrap.cc:492] T 80dac7e5df9248e1bef33b75c3878477 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:48.254953 29613 tablet_bootstrap.cc:654] T 80dac7e5df9248e1bef33b75c3878477 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:48.256631 29614 tablet_bootstrap.cc:654] T adf805b1ae304ebea98cfb2bf3e4eca1 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:48.257464 29616 tablet_bootstrap.cc:492] T f8b10e23797c42c9b3ce5d45a1487f95 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:48.257776 29616 ts_tablet_manager.cc:1367] T f8b10e23797c42c9b3ce5d45a1487f95 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20240501 08:11:48.258947 29616 raft_consensus.cc:348] T f8b10e23797c42c9b3ce5d45a1487f95 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:48.259351 29616 raft_consensus.cc:374] T f8b10e23797c42c9b3ce5d45a1487f95 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:48.259521 29616 raft_consensus.cc:729] T f8b10e23797c42c9b3ce5d45a1487f95 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:48.259852 29616 consensus_queue.cc:260] T f8b10e23797c42c9b3ce5d45a1487f95 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:48.260466 29616 ts_tablet_manager.cc:1398] T f8b10e23797c42c9b3ce5d45a1487f95 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:48.260820 29616 tablet_bootstrap.cc:492] T adf805b1ae304ebea98cfb2bf3e4eca1 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:48.262439 29612 tablet_bootstrap.cc:492] T f8b10e23797c42c9b3ce5d45a1487f95 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:48.262691 29616 tablet_bootstrap.cc:654] T adf805b1ae304ebea98cfb2bf3e4eca1 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:48.262758 29612 ts_tablet_manager.cc:1367] T f8b10e23797c42c9b3ce5d45a1487f95 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.012s
I20240501 08:11:48.263592 29612 raft_consensus.cc:348] T f8b10e23797c42c9b3ce5d45a1487f95 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:48.263875 29612 raft_consensus.cc:374] T f8b10e23797c42c9b3ce5d45a1487f95 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:48.263912 29614 tablet_bootstrap.cc:492] T adf805b1ae304ebea98cfb2bf3e4eca1 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:48.264075 29612 raft_consensus.cc:729] T f8b10e23797c42c9b3ce5d45a1487f95 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:48.264096 29614 ts_tablet_manager.cc:1367] T adf805b1ae304ebea98cfb2bf3e4eca1 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20240501 08:11:48.264380 29612 consensus_queue.cc:260] T f8b10e23797c42c9b3ce5d45a1487f95 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:48.265864 29612 ts_tablet_manager.cc:1398] T f8b10e23797c42c9b3ce5d45a1487f95 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 08:11:48.266220 29612 tablet_bootstrap.cc:492] T adf805b1ae304ebea98cfb2bf3e4eca1 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:48.264951 29614 raft_consensus.cc:348] T adf805b1ae304ebea98cfb2bf3e4eca1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:48.266662 29614 raft_consensus.cc:374] T adf805b1ae304ebea98cfb2bf3e4eca1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:48.266800 29614 raft_consensus.cc:729] T adf805b1ae304ebea98cfb2bf3e4eca1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:48.267140 29614 consensus_queue.cc:260] T adf805b1ae304ebea98cfb2bf3e4eca1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:48.268869 29614 ts_tablet_manager.cc:1398] T adf805b1ae304ebea98cfb2bf3e4eca1 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240501 08:11:48.269258 29614 tablet_bootstrap.cc:492] T f8b10e23797c42c9b3ce5d45a1487f95 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:48.269603 29613 tablet_bootstrap.cc:492] T 80dac7e5df9248e1bef33b75c3878477 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:48.269804 29613 ts_tablet_manager.cc:1367] T 80dac7e5df9248e1bef33b75c3878477 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.007s
I20240501 08:11:48.270678 29613 raft_consensus.cc:348] T 80dac7e5df9248e1bef33b75c3878477 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:48.267923 29612 tablet_bootstrap.cc:654] T adf805b1ae304ebea98cfb2bf3e4eca1 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:48.271246 29614 tablet_bootstrap.cc:654] T f8b10e23797c42c9b3ce5d45a1487f95 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:48.271927 29613 raft_consensus.cc:374] T 80dac7e5df9248e1bef33b75c3878477 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:48.272099 29613 raft_consensus.cc:729] T 80dac7e5df9248e1bef33b75c3878477 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:48.272514 29613 consensus_queue.cc:260] T 80dac7e5df9248e1bef33b75c3878477 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:48.274870 29613 ts_tablet_manager.cc:1398] T 80dac7e5df9248e1bef33b75c3878477 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:11:48.278664 29616 tablet_bootstrap.cc:492] T adf805b1ae304ebea98cfb2bf3e4eca1 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:48.278926 29616 ts_tablet_manager.cc:1367] T adf805b1ae304ebea98cfb2bf3e4eca1 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.007s
I20240501 08:11:48.279233 29614 tablet_bootstrap.cc:492] T f8b10e23797c42c9b3ce5d45a1487f95 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:48.279487 29614 ts_tablet_manager.cc:1367] T f8b10e23797c42c9b3ce5d45a1487f95 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240501 08:11:48.279836 29616 raft_consensus.cc:348] T adf805b1ae304ebea98cfb2bf3e4eca1 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:48.280210 29616 raft_consensus.cc:374] T adf805b1ae304ebea98cfb2bf3e4eca1 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:48.280345 29616 raft_consensus.cc:729] T adf805b1ae304ebea98cfb2bf3e4eca1 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:48.280591 29614 raft_consensus.cc:348] T f8b10e23797c42c9b3ce5d45a1487f95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:48.280714 29616 consensus_queue.cc:260] T adf805b1ae304ebea98cfb2bf3e4eca1 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:48.280982 29614 raft_consensus.cc:374] T f8b10e23797c42c9b3ce5d45a1487f95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:48.281149 29614 raft_consensus.cc:729] T f8b10e23797c42c9b3ce5d45a1487f95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:48.281328 29616 ts_tablet_manager.cc:1398] T adf805b1ae304ebea98cfb2bf3e4eca1 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:48.281500 29614 consensus_queue.cc:260] T f8b10e23797c42c9b3ce5d45a1487f95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:48.281772 29616 tablet_bootstrap.cc:492] T 80dac7e5df9248e1bef33b75c3878477 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:48.284686 29614 ts_tablet_manager.cc:1398] T f8b10e23797c42c9b3ce5d45a1487f95 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 08:11:48.284972 29616 tablet_bootstrap.cc:654] T 80dac7e5df9248e1bef33b75c3878477 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:48.287451 29616 tablet_bootstrap.cc:492] T 80dac7e5df9248e1bef33b75c3878477 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:48.287657 29616 ts_tablet_manager.cc:1367] T 80dac7e5df9248e1bef33b75c3878477 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 08:11:48.288641 29616 raft_consensus.cc:348] T 80dac7e5df9248e1bef33b75c3878477 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:48.288856 29616 raft_consensus.cc:374] T 80dac7e5df9248e1bef33b75c3878477 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:48.289000 29616 raft_consensus.cc:729] T 80dac7e5df9248e1bef33b75c3878477 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:48.289256 29616 consensus_queue.cc:260] T 80dac7e5df9248e1bef33b75c3878477 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:48.289608 29612 tablet_bootstrap.cc:492] T adf805b1ae304ebea98cfb2bf3e4eca1 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:48.289822 29612 ts_tablet_manager.cc:1367] T adf805b1ae304ebea98cfb2bf3e4eca1 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.024s user 0.013s sys 0.004s
I20240501 08:11:48.289968 29616 ts_tablet_manager.cc:1398] T 80dac7e5df9248e1bef33b75c3878477 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:11:48.290647 29612 raft_consensus.cc:348] T adf805b1ae304ebea98cfb2bf3e4eca1 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:48.290891 29612 raft_consensus.cc:374] T adf805b1ae304ebea98cfb2bf3e4eca1 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:48.290994 29612 raft_consensus.cc:729] T adf805b1ae304ebea98cfb2bf3e4eca1 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:48.291256 29612 consensus_queue.cc:260] T adf805b1ae304ebea98cfb2bf3e4eca1 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:48.292560 29612 ts_tablet_manager.cc:1398] T adf805b1ae304ebea98cfb2bf3e4eca1 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:48.292553 29551 raft_consensus.cc:482] T adf805b1ae304ebea98cfb2bf3e4eca1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:48.292719 29551 raft_consensus.cc:504] T adf805b1ae304ebea98cfb2bf3e4eca1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:48.293342 29551 leader_election.cc:288] T adf805b1ae304ebea98cfb2bf3e4eca1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:48.293731 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf805b1ae304ebea98cfb2bf3e4eca1" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:11:48.293924 27889 raft_consensus.cc:2454] T adf805b1ae304ebea98cfb2bf3e4eca1 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:48.294251 27760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf805b1ae304ebea98cfb2bf3e4eca1" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:11:48.294360 28088 leader_election.cc:302] T adf805b1ae304ebea98cfb2bf3e4eca1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:48.294474 27760 raft_consensus.cc:2454] T adf805b1ae304ebea98cfb2bf3e4eca1 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:48.294644 29551 raft_consensus.cc:2781] T adf805b1ae304ebea98cfb2bf3e4eca1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:48.294749 29551 raft_consensus.cc:482] T adf805b1ae304ebea98cfb2bf3e4eca1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:48.294883 29551 raft_consensus.cc:3037] T adf805b1ae304ebea98cfb2bf3e4eca1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:48.297333 29551 raft_consensus.cc:504] T adf805b1ae304ebea98cfb2bf3e4eca1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:48.297844 29551 leader_election.cc:288] T adf805b1ae304ebea98cfb2bf3e4eca1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:48.298308 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf805b1ae304ebea98cfb2bf3e4eca1" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:11:48.298363 27760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf805b1ae304ebea98cfb2bf3e4eca1" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:11:48.298489 27889 raft_consensus.cc:3037] T adf805b1ae304ebea98cfb2bf3e4eca1 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:48.298607 27760 raft_consensus.cc:3037] T adf805b1ae304ebea98cfb2bf3e4eca1 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:48.300796 27889 raft_consensus.cc:2454] T adf805b1ae304ebea98cfb2bf3e4eca1 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:48.300917 27760 raft_consensus.cc:2454] T adf805b1ae304ebea98cfb2bf3e4eca1 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:48.301288 28088 leader_election.cc:302] T adf805b1ae304ebea98cfb2bf3e4eca1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:48.301575 29551 raft_consensus.cc:2781] T adf805b1ae304ebea98cfb2bf3e4eca1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:48.301735 29551 raft_consensus.cc:686] T adf805b1ae304ebea98cfb2bf3e4eca1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:11:48.302045 29551 consensus_queue.cc:237] T adf805b1ae304ebea98cfb2bf3e4eca1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:48.305009 27614 catalog_manager.cc:5484] T adf805b1ae304ebea98cfb2bf3e4eca1 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:48.360277 29551 raft_consensus.cc:482] T 80dac7e5df9248e1bef33b75c3878477 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:48.360474 29551 raft_consensus.cc:504] T 80dac7e5df9248e1bef33b75c3878477 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:48.361413 29551 leader_election.cc:288] T 80dac7e5df9248e1bef33b75c3878477 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:11:48.361856 27760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80dac7e5df9248e1bef33b75c3878477" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:11:48.362015 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80dac7e5df9248e1bef33b75c3878477" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:48.362161 27760 raft_consensus.cc:2454] T 80dac7e5df9248e1bef33b75c3878477 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:48.362269 28021 raft_consensus.cc:2454] T 80dac7e5df9248e1bef33b75c3878477 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:48.362587 28087 leader_election.cc:302] T 80dac7e5df9248e1bef33b75c3878477 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:11:48.362838 29551 raft_consensus.cc:2781] T 80dac7e5df9248e1bef33b75c3878477 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:48.362948 29551 raft_consensus.cc:482] T 80dac7e5df9248e1bef33b75c3878477 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:48.363037 29551 raft_consensus.cc:3037] T 80dac7e5df9248e1bef33b75c3878477 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:48.365074 29551 raft_consensus.cc:504] T 80dac7e5df9248e1bef33b75c3878477 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:48.365324 29550 raft_consensus.cc:482] T f8b10e23797c42c9b3ce5d45a1487f95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:48.365456 29550 raft_consensus.cc:504] T f8b10e23797c42c9b3ce5d45a1487f95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:48.366276 29550 leader_election.cc:288] T f8b10e23797c42c9b3ce5d45a1487f95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:48.366233 27760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80dac7e5df9248e1bef33b75c3878477" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:11:48.366503 27760 raft_consensus.cc:3037] T 80dac7e5df9248e1bef33b75c3878477 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:48.366473 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80dac7e5df9248e1bef33b75c3878477" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:48.366672 28021 raft_consensus.cc:3037] T 80dac7e5df9248e1bef33b75c3878477 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:48.366829 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8b10e23797c42c9b3ce5d45a1487f95" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:11:48.367095 27889 raft_consensus.cc:2454] T f8b10e23797c42c9b3ce5d45a1487f95 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:48.367580 28088 leader_election.cc:302] T f8b10e23797c42c9b3ce5d45a1487f95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:48.367933 29550 raft_consensus.cc:2781] T f8b10e23797c42c9b3ce5d45a1487f95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:48.368052 29550 raft_consensus.cc:482] T f8b10e23797c42c9b3ce5d45a1487f95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:48.368140 29550 raft_consensus.cc:3037] T f8b10e23797c42c9b3ce5d45a1487f95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:48.369418 27760 raft_consensus.cc:2454] T 80dac7e5df9248e1bef33b75c3878477 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:48.369455 28021 raft_consensus.cc:2454] T 80dac7e5df9248e1bef33b75c3878477 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:48.370077 28087 leader_election.cc:302] T 80dac7e5df9248e1bef33b75c3878477 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:48.370087 29551 leader_election.cc:288] T 80dac7e5df9248e1bef33b75c3878477 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:11:48.370389 27760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8b10e23797c42c9b3ce5d45a1487f95" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:11:48.370601 27760 raft_consensus.cc:2454] T f8b10e23797c42c9b3ce5d45a1487f95 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:48.370631 29551 raft_consensus.cc:2781] T 80dac7e5df9248e1bef33b75c3878477 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:48.370903 29550 raft_consensus.cc:504] T f8b10e23797c42c9b3ce5d45a1487f95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:48.371079 29551 raft_consensus.cc:686] T 80dac7e5df9248e1bef33b75c3878477 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:11:48.371398 29551 consensus_queue.cc:237] T 80dac7e5df9248e1bef33b75c3878477 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:48.371630 29550 leader_election.cc:288] T f8b10e23797c42c9b3ce5d45a1487f95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:48.372042 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8b10e23797c42c9b3ce5d45a1487f95" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:11:48.372134 27760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8b10e23797c42c9b3ce5d45a1487f95" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:11:48.372269 27889 raft_consensus.cc:3037] T f8b10e23797c42c9b3ce5d45a1487f95 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:48.372335 27760 raft_consensus.cc:3037] T f8b10e23797c42c9b3ce5d45a1487f95 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:48.374466 27889 raft_consensus.cc:2454] T f8b10e23797c42c9b3ce5d45a1487f95 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:48.374872 28088 leader_election.cc:302] T f8b10e23797c42c9b3ce5d45a1487f95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:48.375139 29551 raft_consensus.cc:2781] T f8b10e23797c42c9b3ce5d45a1487f95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:48.374959 27617 catalog_manager.cc:5484] T 80dac7e5df9248e1bef33b75c3878477 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:48.375308 29551 raft_consensus.cc:686] T f8b10e23797c42c9b3ce5d45a1487f95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:11:48.375547 29551 consensus_queue.cc:237] T f8b10e23797c42c9b3ce5d45a1487f95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:48.375995 27760 raft_consensus.cc:2454] T f8b10e23797c42c9b3ce5d45a1487f95 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:48.385741 27617 catalog_manager.cc:5484] T f8b10e23797c42c9b3ce5d45a1487f95 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:48.695780 29551 consensus_queue.cc:1035] T adf805b1ae304ebea98cfb2bf3e4eca1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:48.702685 29550 consensus_queue.cc:1035] T adf805b1ae304ebea98cfb2bf3e4eca1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:48.764596 29629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:48.764976 29629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:48.772892 29629 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240501 08:11:48.825747 29550 consensus_queue.cc:1035] T f8b10e23797c42c9b3ce5d45a1487f95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:48.842559 29626 consensus_queue.cc:1035] T 80dac7e5df9248e1bef33b75c3878477 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:48.842999 29551 consensus_queue.cc:1035] T f8b10e23797c42c9b3ce5d45a1487f95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:48.883533 29626 consensus_queue.cc:1035] T 80dac7e5df9248e1bef33b75c3878477 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:48.904249 27827 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.131:36305, user_credentials={real_user=slave}} blocked reactor thread for 58075.3us
INSERT report
rows total: 2000
time total: 249.942 ms
time per row: 0.124971 ms
I20240501 08:11:49.173403 27618 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:49.182216 28001 tablet_service.cc:1467] Processing CreateTablet for tablet 0e0f87fa9df24b43bedf063a9063bcc5 (DEFAULT_TABLE table=rebalance_test_table_11 [id=05c446d8d19b4077b3441b0349357c6e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:49.182698 28001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e0f87fa9df24b43bedf063a9063bcc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:49.183233 27739 tablet_service.cc:1467] Processing CreateTablet for tablet 0e0f87fa9df24b43bedf063a9063bcc5 (DEFAULT_TABLE table=rebalance_test_table_11 [id=05c446d8d19b4077b3441b0349357c6e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:49.183641 27739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e0f87fa9df24b43bedf063a9063bcc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:49.183784 27870 tablet_service.cc:1467] Processing CreateTablet for tablet 52b69ecbe4094f32a1ed73b71b934451 (DEFAULT_TABLE table=rebalance_test_table_11 [id=05c446d8d19b4077b3441b0349357c6e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:49.184176 27870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52b69ecbe4094f32a1ed73b71b934451. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:49.184317 27741 tablet_service.cc:1467] Processing CreateTablet for tablet 5eccb9416d7d438893307f13dd4e73dc (DEFAULT_TABLE table=rebalance_test_table_11 [id=05c446d8d19b4077b3441b0349357c6e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:49.184662 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eccb9416d7d438893307f13dd4e73dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:49.187394 27872 tablet_service.cc:1467] Processing CreateTablet for tablet 0e0f87fa9df24b43bedf063a9063bcc5 (DEFAULT_TABLE table=rebalance_test_table_11 [id=05c446d8d19b4077b3441b0349357c6e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:49.187839 27872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e0f87fa9df24b43bedf063a9063bcc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:49.192195 29692 tablet_bootstrap.cc:492] T 0e0f87fa9df24b43bedf063a9063bcc5 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:49.192235 29694 tablet_bootstrap.cc:492] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:49.193977 29692 tablet_bootstrap.cc:654] T 0e0f87fa9df24b43bedf063a9063bcc5 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:49.193979 29694 tablet_bootstrap.cc:654] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:49.194252 29695 tablet_bootstrap.cc:492] T 0e0f87fa9df24b43bedf063a9063bcc5 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:49.195966 29695 tablet_bootstrap.cc:654] T 0e0f87fa9df24b43bedf063a9063bcc5 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:49.196513 28001 tablet_service.cc:1467] Processing CreateTablet for tablet 52b69ecbe4094f32a1ed73b71b934451 (DEFAULT_TABLE table=rebalance_test_table_11 [id=05c446d8d19b4077b3441b0349357c6e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:49.196488 28002 tablet_service.cc:1467] Processing CreateTablet for tablet 5eccb9416d7d438893307f13dd4e73dc (DEFAULT_TABLE table=rebalance_test_table_11 [id=05c446d8d19b4077b3441b0349357c6e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:49.196936 28002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eccb9416d7d438893307f13dd4e73dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:49.197827 28131 tablet_service.cc:1467] Processing CreateTablet for tablet 52b69ecbe4094f32a1ed73b71b934451 (DEFAULT_TABLE table=rebalance_test_table_11 [id=05c446d8d19b4077b3441b0349357c6e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:49.197949 28132 tablet_service.cc:1467] Processing CreateTablet for tablet 5eccb9416d7d438893307f13dd4e73dc (DEFAULT_TABLE table=rebalance_test_table_11 [id=05c446d8d19b4077b3441b0349357c6e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:49.198233 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52b69ecbe4094f32a1ed73b71b934451. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:49.198426 28132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eccb9416d7d438893307f13dd4e73dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:49.202481 28001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52b69ecbe4094f32a1ed73b71b934451. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:49.210681 29698 tablet_bootstrap.cc:492] T 5eccb9416d7d438893307f13dd4e73dc P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:49.216563 29698 tablet_bootstrap.cc:654] T 5eccb9416d7d438893307f13dd4e73dc P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:49.217990 29695 tablet_bootstrap.cc:492] T 0e0f87fa9df24b43bedf063a9063bcc5 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:49.218149 29695 ts_tablet_manager.cc:1367] T 0e0f87fa9df24b43bedf063a9063bcc5 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.008s
I20240501 08:11:49.218966 29694 tablet_bootstrap.cc:492] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:49.219118 29692 tablet_bootstrap.cc:492] T 0e0f87fa9df24b43bedf063a9063bcc5 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:49.219132 29694 ts_tablet_manager.cc:1367] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.027s user 0.017s sys 0.004s
I20240501 08:11:49.219026 29695 raft_consensus.cc:348] T 0e0f87fa9df24b43bedf063a9063bcc5 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:49.219226 29695 raft_consensus.cc:374] T 0e0f87fa9df24b43bedf063a9063bcc5 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:49.219273 29692 ts_tablet_manager.cc:1367] T 0e0f87fa9df24b43bedf063a9063bcc5 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.006s
I20240501 08:11:49.219341 29695 raft_consensus.cc:729] T 0e0f87fa9df24b43bedf063a9063bcc5 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:49.219573 29695 consensus_queue.cc:260] T 0e0f87fa9df24b43bedf063a9063bcc5 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:49.220031 29692 raft_consensus.cc:348] T 0e0f87fa9df24b43bedf063a9063bcc5 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:49.220201 29695 ts_tablet_manager.cc:1398] T 0e0f87fa9df24b43bedf063a9063bcc5 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:11:49.219897 29694 raft_consensus.cc:348] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:49.220430 29694 raft_consensus.cc:374] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:49.220546 29695 tablet_bootstrap.cc:492] T 52b69ecbe4094f32a1ed73b71b934451 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:49.220556 29694 raft_consensus.cc:729] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:49.220918 29698 tablet_bootstrap.cc:492] T 5eccb9416d7d438893307f13dd4e73dc P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:49.220813 29694 consensus_queue.cc:260] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:49.221087 29698 ts_tablet_manager.cc:1367] T 5eccb9416d7d438893307f13dd4e73dc P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.005s
I20240501 08:11:49.221501 29694 ts_tablet_manager.cc:1398] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:49.221784 29694 tablet_bootstrap.cc:492] T 5eccb9416d7d438893307f13dd4e73dc P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:49.222177 29695 tablet_bootstrap.cc:654] T 52b69ecbe4094f32a1ed73b71b934451 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:49.222853 29698 raft_consensus.cc:348] T 5eccb9416d7d438893307f13dd4e73dc P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:49.223098 29692 raft_consensus.cc:374] T 0e0f87fa9df24b43bedf063a9063bcc5 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:49.223223 29692 raft_consensus.cc:729] T 0e0f87fa9df24b43bedf063a9063bcc5 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:49.223381 29694 tablet_bootstrap.cc:654] T 5eccb9416d7d438893307f13dd4e73dc P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:49.223501 29692 consensus_queue.cc:260] T 0e0f87fa9df24b43bedf063a9063bcc5 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:49.223762 29698 raft_consensus.cc:374] T 5eccb9416d7d438893307f13dd4e73dc P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:49.223944 29698 raft_consensus.cc:729] T 5eccb9416d7d438893307f13dd4e73dc P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:49.225479 29692 ts_tablet_manager.cc:1398] T 0e0f87fa9df24b43bedf063a9063bcc5 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240501 08:11:49.225798 29698 consensus_queue.cc:260] T 5eccb9416d7d438893307f13dd4e73dc P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:49.226488 29692 tablet_bootstrap.cc:492] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:49.227089 29698 ts_tablet_manager.cc:1398] T 5eccb9416d7d438893307f13dd4e73dc P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:11:49.229175 29692 tablet_bootstrap.cc:654] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:49.229619 29698 tablet_bootstrap.cc:492] T 52b69ecbe4094f32a1ed73b71b934451 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:49.234365 29694 tablet_bootstrap.cc:492] T 5eccb9416d7d438893307f13dd4e73dc P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:49.234622 29694 ts_tablet_manager.cc:1367] T 5eccb9416d7d438893307f13dd4e73dc P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20240501 08:11:49.234730 29698 tablet_bootstrap.cc:654] T 52b69ecbe4094f32a1ed73b71b934451 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:49.236006 29694 raft_consensus.cc:348] T 5eccb9416d7d438893307f13dd4e73dc P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:49.236315 29694 raft_consensus.cc:374] T 5eccb9416d7d438893307f13dd4e73dc P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:49.236435 29694 raft_consensus.cc:729] T 5eccb9416d7d438893307f13dd4e73dc P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:49.236828 29694 consensus_queue.cc:260] T 5eccb9416d7d438893307f13dd4e73dc P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:49.237433 29694 ts_tablet_manager.cc:1398] T 5eccb9416d7d438893307f13dd4e73dc P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:49.254057 29692 tablet_bootstrap.cc:492] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:49.254182 29698 tablet_bootstrap.cc:492] T 52b69ecbe4094f32a1ed73b71b934451 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:49.254251 29692 ts_tablet_manager.cc:1367] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.008s
I20240501 08:11:49.254328 29695 tablet_bootstrap.cc:492] T 52b69ecbe4094f32a1ed73b71b934451 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:49.254364 29698 ts_tablet_manager.cc:1367] T 52b69ecbe4094f32a1ed73b71b934451 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.013s
I20240501 08:11:49.254496 29695 ts_tablet_manager.cc:1367] T 52b69ecbe4094f32a1ed73b71b934451 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.034s user 0.013s sys 0.020s
I20240501 08:11:49.255482 29698 raft_consensus.cc:348] T 52b69ecbe4094f32a1ed73b71b934451 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:49.255725 29692 raft_consensus.cc:348] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:49.256012 29698 raft_consensus.cc:374] T 52b69ecbe4094f32a1ed73b71b934451 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:49.256268 29692 raft_consensus.cc:374] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:49.256294 29698 raft_consensus.cc:729] T 52b69ecbe4094f32a1ed73b71b934451 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:49.256428 29692 raft_consensus.cc:729] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:49.256822 29698 consensus_queue.cc:260] T 52b69ecbe4094f32a1ed73b71b934451 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:49.257041 29692 consensus_queue.cc:260] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:49.257565 29282 raft_consensus.cc:482] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:49.258067 29698 ts_tablet_manager.cc:1398] T 52b69ecbe4094f32a1ed73b71b934451 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20240501 08:11:49.257907 29282 raft_consensus.cc:504] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:49.258598 29695 raft_consensus.cc:348] T 52b69ecbe4094f32a1ed73b71b934451 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:49.258888 29695 raft_consensus.cc:374] T 52b69ecbe4094f32a1ed73b71b934451 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:49.258975 29695 raft_consensus.cc:729] T 52b69ecbe4094f32a1ed73b71b934451 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:49.260061 29282 leader_election.cc:288] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:11:49.261099 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eccb9416d7d438893307f13dd4e73dc" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:11:49.261214 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eccb9416d7d438893307f13dd4e73dc" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:11:49.261338 28150 raft_consensus.cc:2454] T 5eccb9416d7d438893307f13dd4e73dc P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 0.
I20240501 08:11:49.261451 27759 raft_consensus.cc:2454] T 5eccb9416d7d438893307f13dd4e73dc P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 0.
I20240501 08:11:49.261649 29692 ts_tablet_manager.cc:1398] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20240501 08:11:49.261991 27957 leader_election.cc:302] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:49.263155 29705 raft_consensus.cc:2781] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:49.263622 29705 raft_consensus.cc:482] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:49.263823 29705 raft_consensus.cc:3037] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:49.263729 29695 consensus_queue.cc:260] T 52b69ecbe4094f32a1ed73b71b934451 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:49.263509 29692 tablet_bootstrap.cc:492] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:49.264977 29695 ts_tablet_manager.cc:1398] T 52b69ecbe4094f32a1ed73b71b934451 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240501 08:11:49.266110 29692 tablet_bootstrap.cc:654] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:49.267789 29705 raft_consensus.cc:504] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:49.268829 29705 leader_election.cc:288] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Requested vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:11:49.268960 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eccb9416d7d438893307f13dd4e73dc" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:11:49.269232 27759 raft_consensus.cc:3037] T 5eccb9416d7d438893307f13dd4e73dc P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:49.270866 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eccb9416d7d438893307f13dd4e73dc" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:11:49.271067 28150 raft_consensus.cc:3037] T 5eccb9416d7d438893307f13dd4e73dc P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:49.272347 29692 tablet_bootstrap.cc:492] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:49.272382 27759 raft_consensus.cc:2454] T 5eccb9416d7d438893307f13dd4e73dc P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 1.
I20240501 08:11:49.272501 29692 ts_tablet_manager.cc:1367] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240501 08:11:49.272773 27957 leader_election.cc:302] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:49.273105 29705 raft_consensus.cc:2781] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:49.273270 29705 raft_consensus.cc:686] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Becoming Leader. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Running, Role: LEADER
I20240501 08:11:49.273447 28150 raft_consensus.cc:2454] T 5eccb9416d7d438893307f13dd4e73dc P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 1.
I20240501 08:11:49.273432 29692 raft_consensus.cc:348] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:49.273605 29692 raft_consensus.cc:374] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:49.273555 29705 consensus_queue.cc:237] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:49.273707 29692 raft_consensus.cc:729] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:49.274075 29692 consensus_queue.cc:260] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:49.274710 29692 ts_tablet_manager.cc:1398] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:49.278699 29650 raft_consensus.cc:482] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:49.278851 29650 raft_consensus.cc:504] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:49.279755 29650 leader_election.cc:288] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:11:49.280174 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e0f87fa9df24b43bedf063a9063bcc5" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:49.280267 29556 raft_consensus.cc:482] T 0e0f87fa9df24b43bedf063a9063bcc5 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:49.280395 28022 raft_consensus.cc:2454] T 0e0f87fa9df24b43bedf063a9063bcc5 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:11:49.280434 29556 raft_consensus.cc:504] T 0e0f87fa9df24b43bedf063a9063bcc5 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:49.280406 27618 catalog_manager.cc:5484] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to aad81047009a4ffcbc450b60030583b5 (127.21.84.131). New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:49.280742 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e0f87fa9df24b43bedf063a9063bcc5" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:11:49.280915 27890 raft_consensus.cc:2454] T 0e0f87fa9df24b43bedf063a9063bcc5 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:11:49.281179 29556 leader_election.cc:288] T 0e0f87fa9df24b43bedf063a9063bcc5 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:49.281354 27697 leader_election.cc:302] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:49.281761 29650 raft_consensus.cc:2781] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:49.281868 29650 raft_consensus.cc:482] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:49.281983 29650 raft_consensus.cc:3037] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:49.282373 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e0f87fa9df24b43bedf063a9063bcc5" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:11:49.284283 29650 raft_consensus.cc:504] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:49.284648 27759 raft_consensus.cc:2379] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f1525cbc82a24d13b65f136695c903af in current term 1: Already voted for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in this term.
I20240501 08:11:49.284848 29650 leader_election.cc:288] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Requested vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:11:49.285171 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e0f87fa9df24b43bedf063a9063bcc5" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:49.285334 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e0f87fa9df24b43bedf063a9063bcc5" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:49.285389 28022 raft_consensus.cc:2454] T 0e0f87fa9df24b43bedf063a9063bcc5 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:11:49.285347 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e0f87fa9df24b43bedf063a9063bcc5" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:11:49.285579 27890 raft_consensus.cc:3037] T 0e0f87fa9df24b43bedf063a9063bcc5 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:49.285880 27827 leader_election.cc:302] T 0e0f87fa9df24b43bedf063a9063bcc5 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: aad81047009a4ffcbc450b60030583b5, f1525cbc82a24d13b65f136695c903af; no voters: 6331f77dae9d452184e8cd0cbf5f1b43
I20240501 08:11:49.287953 27890 raft_consensus.cc:2454] T 0e0f87fa9df24b43bedf063a9063bcc5 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:11:49.288100 29556 raft_consensus.cc:2741] T 0e0f87fa9df24b43bedf063a9063bcc5 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240501 08:11:49.288435 27697 leader_election.cc:302] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, f1525cbc82a24d13b65f136695c903af; no voters: aad81047009a4ffcbc450b60030583b5
I20240501 08:11:49.288774 29650 raft_consensus.cc:2781] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:49.288981 29650 raft_consensus.cc:686] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:11:49.289628 29650 consensus_queue.cc:237] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:49.291364 29705 raft_consensus.cc:482] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:49.291509 29705 raft_consensus.cc:504] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:49.292081 29705 leader_election.cc:288] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:11:49.292531 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52b69ecbe4094f32a1ed73b71b934451" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:11:49.292604 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52b69ecbe4094f32a1ed73b71b934451" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:11:49.292764 27890 raft_consensus.cc:2454] T 52b69ecbe4094f32a1ed73b71b934451 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 0.
I20240501 08:11:49.292832 28150 raft_consensus.cc:2454] T 52b69ecbe4094f32a1ed73b71b934451 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 0.
I20240501 08:11:49.292649 27618 catalog_manager.cc:5484] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:49.293186 27958 leader_election.cc:302] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aad81047009a4ffcbc450b60030583b5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:49.293426 29705 raft_consensus.cc:2781] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:49.293556 29705 raft_consensus.cc:482] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:49.293673 29705 raft_consensus.cc:3037] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:49.296177 29705 raft_consensus.cc:504] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:49.296766 29705 leader_election.cc:288] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Requested vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:11:49.297209 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52b69ecbe4094f32a1ed73b71b934451" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:11:49.297386 27890 raft_consensus.cc:3037] T 52b69ecbe4094f32a1ed73b71b934451 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:49.297374 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52b69ecbe4094f32a1ed73b71b934451" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:11:49.297611 28150 raft_consensus.cc:3037] T 52b69ecbe4094f32a1ed73b71b934451 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:49.299459 27890 raft_consensus.cc:2454] T 52b69ecbe4094f32a1ed73b71b934451 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 1.
I20240501 08:11:49.299510 28150 raft_consensus.cc:2454] T 52b69ecbe4094f32a1ed73b71b934451 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 1.
I20240501 08:11:49.299922 27958 leader_election.cc:302] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aad81047009a4ffcbc450b60030583b5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:49.300222 29705 raft_consensus.cc:2781] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:49.300415 29705 raft_consensus.cc:686] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Becoming Leader. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Running, Role: LEADER
I20240501 08:11:49.300760 29705 consensus_queue.cc:237] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:49.303815 27618 catalog_manager.cc:5484] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to aad81047009a4ffcbc450b60030583b5 (127.21.84.131). New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } }
W20240501 08:11:49.675133 29711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:49.675511 29711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:49.683714 29711 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:11:49.697985 29534 consensus_queue.cc:1035] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:49.704619 28022 raft_consensus.cc:3037] T 0e0f87fa9df24b43bedf063a9063bcc5 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:49.713229 29725 consensus_queue.cc:1035] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:49.731014 29705 consensus_queue.cc:1035] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:49.749807 29282 consensus_queue.cc:1035] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:49.753434 29282 consensus_queue.cc:1035] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_11'
I20240501 08:11:49.775525 29706 consensus_queue.cc:1035] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 344.929 ms
time per row: 0.172464 ms
I20240501 08:11:50.274484 27613 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalance_test_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:50.283870 27739 tablet_service.cc:1467] Processing CreateTablet for tablet d3b2e8d327844b9ba2f639872c2981e7 (DEFAULT_TABLE table=rebalance_test_table_12 [id=f662392c9baa4a02b6a9549ded7caf1b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:50.284379 27739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3b2e8d327844b9ba2f639872c2981e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:50.285902 28131 tablet_service.cc:1467] Processing CreateTablet for tablet 9ba9a204ac9a4c5b8c7bd8c776f49db8 (DEFAULT_TABLE table=rebalance_test_table_12 [id=f662392c9baa4a02b6a9549ded7caf1b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:50.286362 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ba9a204ac9a4c5b8c7bd8c776f49db8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:50.286365 27872 tablet_service.cc:1467] Processing CreateTablet for tablet d3b2e8d327844b9ba2f639872c2981e7 (DEFAULT_TABLE table=rebalance_test_table_12 [id=f662392c9baa4a02b6a9549ded7caf1b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:50.286751 27872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3b2e8d327844b9ba2f639872c2981e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:50.287153 27871 tablet_service.cc:1467] Processing CreateTablet for tablet a1a23302b0ad4101b8095fbf647e65ef (DEFAULT_TABLE table=rebalance_test_table_12 [id=f662392c9baa4a02b6a9549ded7caf1b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:50.287540 27871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1a23302b0ad4101b8095fbf647e65ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:50.290387 28001 tablet_service.cc:1467] Processing CreateTablet for tablet d3b2e8d327844b9ba2f639872c2981e7 (DEFAULT_TABLE table=rebalance_test_table_12 [id=f662392c9baa4a02b6a9549ded7caf1b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:50.290822 28001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3b2e8d327844b9ba2f639872c2981e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:50.294502 27870 tablet_service.cc:1467] Processing CreateTablet for tablet 9ba9a204ac9a4c5b8c7bd8c776f49db8 (DEFAULT_TABLE table=rebalance_test_table_12 [id=f662392c9baa4a02b6a9549ded7caf1b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:50.294509 28002 tablet_service.cc:1467] Processing CreateTablet for tablet a1a23302b0ad4101b8095fbf647e65ef (DEFAULT_TABLE table=rebalance_test_table_12 [id=f662392c9baa4a02b6a9549ded7caf1b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:50.295025 28002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1a23302b0ad4101b8095fbf647e65ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:50.295048 27870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ba9a204ac9a4c5b8c7bd8c776f49db8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:50.298034 28132 tablet_service.cc:1467] Processing CreateTablet for tablet a1a23302b0ad4101b8095fbf647e65ef (DEFAULT_TABLE table=rebalance_test_table_12 [id=f662392c9baa4a02b6a9549ded7caf1b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:50.298691 28132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1a23302b0ad4101b8095fbf647e65ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:50.302261 29771 tablet_bootstrap.cc:492] T d3b2e8d327844b9ba2f639872c2981e7 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:50.303205 27741 tablet_service.cc:1467] Processing CreateTablet for tablet 9ba9a204ac9a4c5b8c7bd8c776f49db8 (DEFAULT_TABLE table=rebalance_test_table_12 [id=f662392c9baa4a02b6a9549ded7caf1b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:50.303800 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ba9a204ac9a4c5b8c7bd8c776f49db8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:50.304219 29771 tablet_bootstrap.cc:654] T d3b2e8d327844b9ba2f639872c2981e7 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:50.325403 29774 tablet_bootstrap.cc:492] T a1a23302b0ad4101b8095fbf647e65ef P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:50.325493 29773 tablet_bootstrap.cc:492] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:50.327783 29774 tablet_bootstrap.cc:654] T a1a23302b0ad4101b8095fbf647e65ef P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:50.327852 29773 tablet_bootstrap.cc:654] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:50.330299 29770 tablet_bootstrap.cc:492] T d3b2e8d327844b9ba2f639872c2981e7 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:50.331058 29774 tablet_bootstrap.cc:492] T a1a23302b0ad4101b8095fbf647e65ef P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:50.331210 29774 ts_tablet_manager.cc:1367] T a1a23302b0ad4101b8095fbf647e65ef P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240501 08:11:50.331969 29770 tablet_bootstrap.cc:654] T d3b2e8d327844b9ba2f639872c2981e7 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:50.331986 29774 raft_consensus.cc:348] T a1a23302b0ad4101b8095fbf647e65ef P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:50.332304 29774 raft_consensus.cc:374] T a1a23302b0ad4101b8095fbf647e65ef P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:50.332417 29774 raft_consensus.cc:729] T a1a23302b0ad4101b8095fbf647e65ef P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:50.332633 29774 consensus_queue.cc:260] T a1a23302b0ad4101b8095fbf647e65ef P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:50.333740 29771 tablet_bootstrap.cc:492] T d3b2e8d327844b9ba2f639872c2981e7 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:50.333899 29771 ts_tablet_manager.cc:1367] T d3b2e8d327844b9ba2f639872c2981e7 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.018s
I20240501 08:11:50.334709 29771 raft_consensus.cc:348] T d3b2e8d327844b9ba2f639872c2981e7 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:50.334903 29771 raft_consensus.cc:374] T d3b2e8d327844b9ba2f639872c2981e7 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:50.335002 29771 raft_consensus.cc:729] T d3b2e8d327844b9ba2f639872c2981e7 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:50.335219 29774 ts_tablet_manager.cc:1398] T a1a23302b0ad4101b8095fbf647e65ef P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20240501 08:11:50.335208 29771 consensus_queue.cc:260] T d3b2e8d327844b9ba2f639872c2981e7 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:50.335497 29774 tablet_bootstrap.cc:492] T d3b2e8d327844b9ba2f639872c2981e7 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:50.335712 29771 ts_tablet_manager.cc:1398] T d3b2e8d327844b9ba2f639872c2981e7 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:11:50.335961 29771 tablet_bootstrap.cc:492] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:50.337260 29774 tablet_bootstrap.cc:654] T d3b2e8d327844b9ba2f639872c2981e7 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:50.337545 29771 tablet_bootstrap.cc:654] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:50.341187 29773 tablet_bootstrap.cc:492] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:50.341398 29773 ts_tablet_manager.cc:1367] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20240501 08:11:50.346187 29773 raft_consensus.cc:348] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:50.347522 29773 raft_consensus.cc:374] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:50.347730 29773 raft_consensus.cc:729] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:50.348349 29773 consensus_queue.cc:260] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:50.349289 29773 ts_tablet_manager.cc:1398] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240501 08:11:50.349658 29773 tablet_bootstrap.cc:492] T a1a23302b0ad4101b8095fbf647e65ef P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:50.349751 29774 tablet_bootstrap.cc:492] T d3b2e8d327844b9ba2f639872c2981e7 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:50.350421 29774 ts_tablet_manager.cc:1367] T d3b2e8d327844b9ba2f639872c2981e7 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.006s
I20240501 08:11:50.351634 29774 raft_consensus.cc:348] T d3b2e8d327844b9ba2f639872c2981e7 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:50.352048 29774 raft_consensus.cc:374] T d3b2e8d327844b9ba2f639872c2981e7 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:50.352191 29774 raft_consensus.cc:729] T d3b2e8d327844b9ba2f639872c2981e7 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:50.352627 29774 consensus_queue.cc:260] T d3b2e8d327844b9ba2f639872c2981e7 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:50.353996 29771 tablet_bootstrap.cc:492] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:50.354249 29771 ts_tablet_manager.cc:1367] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.003s
I20240501 08:11:50.354815 29774 ts_tablet_manager.cc:1398] T d3b2e8d327844b9ba2f639872c2981e7 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:11:50.355134 29771 raft_consensus.cc:348] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:50.355340 29771 raft_consensus.cc:374] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:50.355424 29771 raft_consensus.cc:729] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:50.355650 29771 consensus_queue.cc:260] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:50.356195 29771 ts_tablet_manager.cc:1398] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:11:50.356717 29771 tablet_bootstrap.cc:492] T a1a23302b0ad4101b8095fbf647e65ef P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:50.358875 29771 tablet_bootstrap.cc:654] T a1a23302b0ad4101b8095fbf647e65ef P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:50.367338 29773 tablet_bootstrap.cc:654] T a1a23302b0ad4101b8095fbf647e65ef P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:50.368273 29771 tablet_bootstrap.cc:492] T a1a23302b0ad4101b8095fbf647e65ef P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:50.368505 29771 ts_tablet_manager.cc:1367] T a1a23302b0ad4101b8095fbf647e65ef P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.002s
I20240501 08:11:50.369479 29771 raft_consensus.cc:348] T a1a23302b0ad4101b8095fbf647e65ef P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:50.369835 29771 raft_consensus.cc:374] T a1a23302b0ad4101b8095fbf647e65ef P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:50.370013 29771 raft_consensus.cc:729] T a1a23302b0ad4101b8095fbf647e65ef P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:50.370342 29771 consensus_queue.cc:260] T a1a23302b0ad4101b8095fbf647e65ef P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:50.371234 29771 ts_tablet_manager.cc:1398] T a1a23302b0ad4101b8095fbf647e65ef P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:50.376324 29773 tablet_bootstrap.cc:492] T a1a23302b0ad4101b8095fbf647e65ef P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:50.376405 29770 tablet_bootstrap.cc:492] T d3b2e8d327844b9ba2f639872c2981e7 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:50.376531 29773 ts_tablet_manager.cc:1367] T a1a23302b0ad4101b8095fbf647e65ef P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20240501 08:11:50.376538 29770 ts_tablet_manager.cc:1367] T d3b2e8d327844b9ba2f639872c2981e7 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.046s user 0.012s sys 0.031s
I20240501 08:11:50.377375 29773 raft_consensus.cc:348] T a1a23302b0ad4101b8095fbf647e65ef P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:50.377714 29773 raft_consensus.cc:374] T a1a23302b0ad4101b8095fbf647e65ef P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:50.377813 29773 raft_consensus.cc:729] T a1a23302b0ad4101b8095fbf647e65ef P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:50.377732 29770 raft_consensus.cc:348] T d3b2e8d327844b9ba2f639872c2981e7 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:50.378036 29770 raft_consensus.cc:374] T d3b2e8d327844b9ba2f639872c2981e7 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:50.378098 29773 consensus_queue.cc:260] T a1a23302b0ad4101b8095fbf647e65ef P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:50.378175 29770 raft_consensus.cc:729] T d3b2e8d327844b9ba2f639872c2981e7 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:50.378499 29770 consensus_queue.cc:260] T d3b2e8d327844b9ba2f639872c2981e7 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:50.379576 29773 ts_tablet_manager.cc:1398] T a1a23302b0ad4101b8095fbf647e65ef P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 08:11:50.379891 29770 ts_tablet_manager.cc:1398] T d3b2e8d327844b9ba2f639872c2981e7 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240501 08:11:50.380134 29770 tablet_bootstrap.cc:492] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:50.381451 29770 tablet_bootstrap.cc:654] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:50.383683 29770 tablet_bootstrap.cc:492] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:50.383867 29770 ts_tablet_manager.cc:1367] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 08:11:50.384658 29770 raft_consensus.cc:348] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:50.384835 29770 raft_consensus.cc:374] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:50.384888 29770 raft_consensus.cc:729] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:50.385087 29770 consensus_queue.cc:260] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:50.385486 29770 ts_tablet_manager.cc:1398] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:11:50.448267 29705 raft_consensus.cc:482] T a1a23302b0ad4101b8095fbf647e65ef P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:50.448495 29705 raft_consensus.cc:504] T a1a23302b0ad4101b8095fbf647e65ef P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:50.449120 29705 leader_election.cc:288] T a1a23302b0ad4101b8095fbf647e65ef P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:11:50.449705 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a23302b0ad4101b8095fbf647e65ef" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:11:50.449709 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a23302b0ad4101b8095fbf647e65ef" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:11:50.449965 28151 raft_consensus.cc:2454] T a1a23302b0ad4101b8095fbf647e65ef P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 0.
I20240501 08:11:50.449978 27890 raft_consensus.cc:2454] T a1a23302b0ad4101b8095fbf647e65ef P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 0.
I20240501 08:11:50.450474 27955 leader_election.cc:302] T a1a23302b0ad4101b8095fbf647e65ef P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, aad81047009a4ffcbc450b60030583b5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:50.450819 29705 raft_consensus.cc:2781] T a1a23302b0ad4101b8095fbf647e65ef P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:50.450928 29705 raft_consensus.cc:482] T a1a23302b0ad4101b8095fbf647e65ef P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:50.451025 29705 raft_consensus.cc:3037] T a1a23302b0ad4101b8095fbf647e65ef P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:50.453085 29705 raft_consensus.cc:504] T a1a23302b0ad4101b8095fbf647e65ef P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:50.453770 29705 leader_election.cc:288] T a1a23302b0ad4101b8095fbf647e65ef P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Requested vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:11:50.454272 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a23302b0ad4101b8095fbf647e65ef" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:11:50.454281 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a23302b0ad4101b8095fbf647e65ef" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:11:50.454493 27889 raft_consensus.cc:3037] T a1a23302b0ad4101b8095fbf647e65ef P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:50.454520 28151 raft_consensus.cc:3037] T a1a23302b0ad4101b8095fbf647e65ef P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:50.456383 27889 raft_consensus.cc:2454] T a1a23302b0ad4101b8095fbf647e65ef P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 1.
I20240501 08:11:50.456470 28151 raft_consensus.cc:2454] T a1a23302b0ad4101b8095fbf647e65ef P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 1.
I20240501 08:11:50.456907 27958 leader_election.cc:302] T a1a23302b0ad4101b8095fbf647e65ef P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aad81047009a4ffcbc450b60030583b5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:50.457767 29705 raft_consensus.cc:2781] T a1a23302b0ad4101b8095fbf647e65ef P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:50.458096 29705 raft_consensus.cc:686] T a1a23302b0ad4101b8095fbf647e65ef P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Becoming Leader. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Running, Role: LEADER
I20240501 08:11:50.458413 29705 consensus_queue.cc:237] T a1a23302b0ad4101b8095fbf647e65ef P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:50.461957 27617 catalog_manager.cc:5484] T a1a23302b0ad4101b8095fbf647e65ef P aad81047009a4ffcbc450b60030583b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to aad81047009a4ffcbc450b60030583b5 (127.21.84.131). New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:50.469357 29705 raft_consensus.cc:482] T d3b2e8d327844b9ba2f639872c2981e7 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:50.469570 29705 raft_consensus.cc:504] T d3b2e8d327844b9ba2f639872c2981e7 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:50.470229 29705 leader_election.cc:288] T d3b2e8d327844b9ba2f639872c2981e7 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:11:50.470666 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b2e8d327844b9ba2f639872c2981e7" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:11:50.470772 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b2e8d327844b9ba2f639872c2981e7" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:11:50.470883 27759 raft_consensus.cc:2454] T d3b2e8d327844b9ba2f639872c2981e7 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 0.
I20240501 08:11:50.471025 27889 raft_consensus.cc:2454] T d3b2e8d327844b9ba2f639872c2981e7 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 0.
I20240501 08:11:50.471304 27957 leader_election.cc:302] T d3b2e8d327844b9ba2f639872c2981e7 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:50.471594 29705 raft_consensus.cc:2781] T d3b2e8d327844b9ba2f639872c2981e7 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:50.471675 29705 raft_consensus.cc:482] T d3b2e8d327844b9ba2f639872c2981e7 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:50.471758 29705 raft_consensus.cc:3037] T d3b2e8d327844b9ba2f639872c2981e7 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:50.473650 29705 raft_consensus.cc:504] T d3b2e8d327844b9ba2f639872c2981e7 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:50.474241 29705 leader_election.cc:288] T d3b2e8d327844b9ba2f639872c2981e7 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Requested vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:11:50.474764 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b2e8d327844b9ba2f639872c2981e7" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:11:50.474879 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b2e8d327844b9ba2f639872c2981e7" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:11:50.475021 27759 raft_consensus.cc:3037] T d3b2e8d327844b9ba2f639872c2981e7 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:50.475139 27889 raft_consensus.cc:3037] T d3b2e8d327844b9ba2f639872c2981e7 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:50.477949 27759 raft_consensus.cc:2454] T d3b2e8d327844b9ba2f639872c2981e7 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 1.
I20240501 08:11:50.477967 27889 raft_consensus.cc:2454] T d3b2e8d327844b9ba2f639872c2981e7 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 1.
I20240501 08:11:50.478385 27958 leader_election.cc:302] T d3b2e8d327844b9ba2f639872c2981e7 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aad81047009a4ffcbc450b60030583b5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:50.478729 29705 raft_consensus.cc:2781] T d3b2e8d327844b9ba2f639872c2981e7 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:50.478914 29705 raft_consensus.cc:686] T d3b2e8d327844b9ba2f639872c2981e7 P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Becoming Leader. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Running, Role: LEADER
I20240501 08:11:50.479180 29705 consensus_queue.cc:237] T d3b2e8d327844b9ba2f639872c2981e7 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:50.479363 29670 raft_consensus.cc:482] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:50.479573 29670 raft_consensus.cc:504] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:50.480926 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ba9a204ac9a4c5b8c7bd8c776f49db8" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:11:50.480933 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ba9a204ac9a4c5b8c7bd8c776f49db8" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:11:50.481144 27759 raft_consensus.cc:2454] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:50.481215 27889 raft_consensus.cc:2454] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:50.481581 28087 leader_election.cc:302] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:11:50.481904 29784 raft_consensus.cc:2781] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:50.482077 29784 raft_consensus.cc:482] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:50.482220 29784 raft_consensus.cc:3037] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:50.482478 29670 leader_election.cc:288] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:50.483953 27617 catalog_manager.cc:5484] T d3b2e8d327844b9ba2f639872c2981e7 P aad81047009a4ffcbc450b60030583b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to aad81047009a4ffcbc450b60030583b5 (127.21.84.131). New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:50.485023 29784 raft_consensus.cc:504] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:50.485921 29784 leader_election.cc:288] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:50.486117 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ba9a204ac9a4c5b8c7bd8c776f49db8" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:11:50.486323 27889 raft_consensus.cc:3037] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:50.488786 27889 raft_consensus.cc:2454] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:50.489307 28088 leader_election.cc:302] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:50.489734 29784 raft_consensus.cc:2781] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:50.489866 29784 raft_consensus.cc:686] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:11:50.490140 29784 consensus_queue.cc:237] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:50.492640 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ba9a204ac9a4c5b8c7bd8c776f49db8" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:11:50.492825 27759 raft_consensus.cc:3037] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:50.493630 27617 catalog_manager.cc:5484] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:50.494835 27759 raft_consensus.cc:2454] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
W20240501 08:11:50.867815 29788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:50.868162 29788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:50.876276 29788 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:11:50.902211 29705 consensus_queue.cc:1035] T d3b2e8d327844b9ba2f639872c2981e7 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:50.908329 29705 consensus_queue.cc:1035] T d3b2e8d327844b9ba2f639872c2981e7 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_12'
I20240501 08:11:50.989063 29786 consensus_queue.cc:1035] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:51.041249 29784 consensus_queue.cc:1035] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:51.045982 29705 consensus_queue.cc:1035] T a1a23302b0ad4101b8095fbf647e65ef P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:51.074323 29705 consensus_queue.cc:1035] T a1a23302b0ad4101b8095fbf647e65ef P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 334.186 ms
time per row: 0.167093 ms
I20240501 08:11:51.375916 27617 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalance_test_table_13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:51.384958 27739 tablet_service.cc:1467] Processing CreateTablet for tablet f3b6691cbe8740d2978c20209c78f3ff (DEFAULT_TABLE table=rebalance_test_table_13 [id=86d1b715252040a3bc0f714b3db3f4ff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:51.385493 27739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3b6691cbe8740d2978c20209c78f3ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:51.385672 27741 tablet_service.cc:1467] Processing CreateTablet for tablet bb7dae019c7e4842b0daa35ee4b40367 (DEFAULT_TABLE table=rebalance_test_table_13 [id=86d1b715252040a3bc0f714b3db3f4ff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:51.386417 27870 tablet_service.cc:1467] Processing CreateTablet for tablet 9820951bffc749139229e2b79c8ab597 (DEFAULT_TABLE table=rebalance_test_table_13 [id=86d1b715252040a3bc0f714b3db3f4ff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:51.386709 27740 tablet_service.cc:1467] Processing CreateTablet for tablet 9820951bffc749139229e2b79c8ab597 (DEFAULT_TABLE table=rebalance_test_table_13 [id=86d1b715252040a3bc0f714b3db3f4ff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:51.386727 28001 tablet_service.cc:1467] Processing CreateTablet for tablet bb7dae019c7e4842b0daa35ee4b40367 (DEFAULT_TABLE table=rebalance_test_table_13 [id=86d1b715252040a3bc0f714b3db3f4ff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:51.386832 27870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9820951bffc749139229e2b79c8ab597. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:51.387104 28001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb7dae019c7e4842b0daa35ee4b40367. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:51.387111 27740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9820951bffc749139229e2b79c8ab597. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:51.390400 28002 tablet_service.cc:1467] Processing CreateTablet for tablet f3b6691cbe8740d2978c20209c78f3ff (DEFAULT_TABLE table=rebalance_test_table_13 [id=86d1b715252040a3bc0f714b3db3f4ff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:51.390901 28002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3b6691cbe8740d2978c20209c78f3ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:51.392748 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb7dae019c7e4842b0daa35ee4b40367. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:51.394527 28131 tablet_service.cc:1467] Processing CreateTablet for tablet f3b6691cbe8740d2978c20209c78f3ff (DEFAULT_TABLE table=rebalance_test_table_13 [id=86d1b715252040a3bc0f714b3db3f4ff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:51.395076 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3b6691cbe8740d2978c20209c78f3ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:51.398509 28132 tablet_service.cc:1467] Processing CreateTablet for tablet bb7dae019c7e4842b0daa35ee4b40367 (DEFAULT_TABLE table=rebalance_test_table_13 [id=86d1b715252040a3bc0f714b3db3f4ff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:51.399019 28132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb7dae019c7e4842b0daa35ee4b40367. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:51.404191 28000 tablet_service.cc:1467] Processing CreateTablet for tablet 9820951bffc749139229e2b79c8ab597 (DEFAULT_TABLE table=rebalance_test_table_13 [id=86d1b715252040a3bc0f714b3db3f4ff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:51.404654 28000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9820951bffc749139229e2b79c8ab597. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:51.405330 29855 tablet_bootstrap.cc:492] T bb7dae019c7e4842b0daa35ee4b40367 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:51.408145 29855 tablet_bootstrap.cc:654] T bb7dae019c7e4842b0daa35ee4b40367 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:51.409281 29854 tablet_bootstrap.cc:492] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:51.411007 29854 tablet_bootstrap.cc:654] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:51.412415 29855 tablet_bootstrap.cc:492] T bb7dae019c7e4842b0daa35ee4b40367 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:51.412626 29855 ts_tablet_manager.cc:1367] T bb7dae019c7e4842b0daa35ee4b40367 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 08:11:51.413429 29854 tablet_bootstrap.cc:492] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:51.413398 29855 raft_consensus.cc:348] T bb7dae019c7e4842b0daa35ee4b40367 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:51.413650 29854 ts_tablet_manager.cc:1367] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240501 08:11:51.413689 29855 raft_consensus.cc:374] T bb7dae019c7e4842b0daa35ee4b40367 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:51.413825 29855 raft_consensus.cc:729] T bb7dae019c7e4842b0daa35ee4b40367 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:51.414187 29855 consensus_queue.cc:260] T bb7dae019c7e4842b0daa35ee4b40367 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:51.414891 29855 ts_tablet_manager.cc:1398] T bb7dae019c7e4842b0daa35ee4b40367 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:51.414501 29854 raft_consensus.cc:348] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:51.415561 29859 tablet_bootstrap.cc:492] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:51.415748 29854 raft_consensus.cc:374] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:51.415935 29854 raft_consensus.cc:729] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:51.416245 29854 consensus_queue.cc:260] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:51.416877 29854 ts_tablet_manager.cc:1398] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:11:51.418061 29854 tablet_bootstrap.cc:492] T 9820951bffc749139229e2b79c8ab597 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:51.418256 29859 tablet_bootstrap.cc:654] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:51.419701 29854 tablet_bootstrap.cc:654] T 9820951bffc749139229e2b79c8ab597 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:51.420007 29857 tablet_bootstrap.cc:492] T f3b6691cbe8740d2978c20209c78f3ff P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:51.422885 29857 tablet_bootstrap.cc:654] T f3b6691cbe8740d2978c20209c78f3ff P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:51.428865 29854 tablet_bootstrap.cc:492] T 9820951bffc749139229e2b79c8ab597 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:51.429147 29854 ts_tablet_manager.cc:1367] T 9820951bffc749139229e2b79c8ab597 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240501 08:11:51.430048 29854 raft_consensus.cc:348] T 9820951bffc749139229e2b79c8ab597 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:51.430388 29854 raft_consensus.cc:374] T 9820951bffc749139229e2b79c8ab597 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:51.430516 29854 raft_consensus.cc:729] T 9820951bffc749139229e2b79c8ab597 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:51.430876 29854 consensus_queue.cc:260] T 9820951bffc749139229e2b79c8ab597 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:51.431855 29854 ts_tablet_manager.cc:1398] T 9820951bffc749139229e2b79c8ab597 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:51.432226 29855 tablet_bootstrap.cc:492] T 9820951bffc749139229e2b79c8ab597 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:51.433248 29854 tablet_bootstrap.cc:492] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:51.434967 29854 tablet_bootstrap.cc:654] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:51.435910 29857 tablet_bootstrap.cc:492] T f3b6691cbe8740d2978c20209c78f3ff P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:51.436115 29857 ts_tablet_manager.cc:1367] T f3b6691cbe8740d2978c20209c78f3ff P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.007s
I20240501 08:11:51.437135 29857 raft_consensus.cc:348] T f3b6691cbe8740d2978c20209c78f3ff P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:51.437418 29857 raft_consensus.cc:374] T f3b6691cbe8740d2978c20209c78f3ff P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:51.437536 29857 raft_consensus.cc:729] T f3b6691cbe8740d2978c20209c78f3ff P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:51.437825 29857 consensus_queue.cc:260] T f3b6691cbe8740d2978c20209c78f3ff P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:51.441130 29857 ts_tablet_manager.cc:1398] T f3b6691cbe8740d2978c20209c78f3ff P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 08:11:51.443135 29855 tablet_bootstrap.cc:654] T 9820951bffc749139229e2b79c8ab597 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:51.443769 29857 tablet_bootstrap.cc:492] T bb7dae019c7e4842b0daa35ee4b40367 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:51.444794 29854 tablet_bootstrap.cc:492] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:51.445025 29854 ts_tablet_manager.cc:1367] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20240501 08:11:51.445987 29854 raft_consensus.cc:348] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:51.446347 29854 raft_consensus.cc:374] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:51.446486 29854 raft_consensus.cc:729] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:51.446789 29854 consensus_queue.cc:260] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:51.448676 29854 ts_tablet_manager.cc:1398] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:11:51.450618 29857 tablet_bootstrap.cc:654] T bb7dae019c7e4842b0daa35ee4b40367 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:51.451825 29859 tablet_bootstrap.cc:492] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:51.452117 29859 ts_tablet_manager.cc:1367] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.037s user 0.008s sys 0.023s
I20240501 08:11:51.452589 29855 tablet_bootstrap.cc:492] T 9820951bffc749139229e2b79c8ab597 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:51.452786 29855 ts_tablet_manager.cc:1367] T 9820951bffc749139229e2b79c8ab597 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.004s
I20240501 08:11:51.453076 29859 raft_consensus.cc:348] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:51.453305 29859 raft_consensus.cc:374] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:51.453477 29859 raft_consensus.cc:729] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:51.453454 29855 raft_consensus.cc:348] T 9820951bffc749139229e2b79c8ab597 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:51.453759 29855 raft_consensus.cc:374] T 9820951bffc749139229e2b79c8ab597 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:51.453878 29855 raft_consensus.cc:729] T 9820951bffc749139229e2b79c8ab597 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:51.453830 29859 consensus_queue.cc:260] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:51.454434 29855 consensus_queue.cc:260] T 9820951bffc749139229e2b79c8ab597 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:51.457090 29859 ts_tablet_manager.cc:1398] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 08:11:51.458559 29855 ts_tablet_manager.cc:1398] T 9820951bffc749139229e2b79c8ab597 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240501 08:11:51.458892 29855 tablet_bootstrap.cc:492] T f3b6691cbe8740d2978c20209c78f3ff P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:51.460310 29855 tablet_bootstrap.cc:654] T f3b6691cbe8740d2978c20209c78f3ff P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:51.460974 29857 tablet_bootstrap.cc:492] T bb7dae019c7e4842b0daa35ee4b40367 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:51.461133 29857 ts_tablet_manager.cc:1367] T bb7dae019c7e4842b0daa35ee4b40367 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.003s
I20240501 08:11:51.462023 29857 raft_consensus.cc:348] T bb7dae019c7e4842b0daa35ee4b40367 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:51.462345 29857 raft_consensus.cc:374] T bb7dae019c7e4842b0daa35ee4b40367 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:51.462477 29857 raft_consensus.cc:729] T bb7dae019c7e4842b0daa35ee4b40367 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:51.462615 29855 tablet_bootstrap.cc:492] T f3b6691cbe8740d2978c20209c78f3ff P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:51.462779 29855 ts_tablet_manager.cc:1367] T f3b6691cbe8740d2978c20209c78f3ff P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 08:11:51.462869 29857 consensus_queue.cc:260] T bb7dae019c7e4842b0daa35ee4b40367 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:51.463577 29855 raft_consensus.cc:348] T f3b6691cbe8740d2978c20209c78f3ff P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:51.463800 29855 raft_consensus.cc:374] T f3b6691cbe8740d2978c20209c78f3ff P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:51.463912 29855 raft_consensus.cc:729] T f3b6691cbe8740d2978c20209c78f3ff P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:51.464133 29855 consensus_queue.cc:260] T f3b6691cbe8740d2978c20209c78f3ff P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:51.464601 29857 ts_tablet_manager.cc:1398] T bb7dae019c7e4842b0daa35ee4b40367 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:11:51.464658 29855 ts_tablet_manager.cc:1398] T f3b6691cbe8740d2978c20209c78f3ff P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:11:51.472262 29650 raft_consensus.cc:482] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:51.472445 29650 raft_consensus.cc:504] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:51.473105 29650 leader_election.cc:288] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:11:51.473578 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3b6691cbe8740d2978c20209c78f3ff" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:51.473600 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3b6691cbe8740d2978c20209c78f3ff" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:11:51.473768 28022 raft_consensus.cc:2454] T f3b6691cbe8740d2978c20209c78f3ff P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:11:51.473795 28150 raft_consensus.cc:2454] T f3b6691cbe8740d2978c20209c78f3ff P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:11:51.474260 27696 leader_election.cc:302] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:51.474599 29650 raft_consensus.cc:2781] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:51.474682 29650 raft_consensus.cc:482] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:51.474803 29650 raft_consensus.cc:3037] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:51.476704 29650 raft_consensus.cc:504] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:51.477430 29650 leader_election.cc:288] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Requested vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:11:51.477821 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3b6691cbe8740d2978c20209c78f3ff" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:51.478066 28022 raft_consensus.cc:3037] T f3b6691cbe8740d2978c20209c78f3ff P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:51.478066 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3b6691cbe8740d2978c20209c78f3ff" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:11:51.478253 28150 raft_consensus.cc:3037] T f3b6691cbe8740d2978c20209c78f3ff P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:51.479943 28022 raft_consensus.cc:2454] T f3b6691cbe8740d2978c20209c78f3ff P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:11:51.480068 28150 raft_consensus.cc:2454] T f3b6691cbe8740d2978c20209c78f3ff P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:11:51.480419 27694 leader_election.cc:302] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:11:51.480685 29650 raft_consensus.cc:2781] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:51.480825 29650 raft_consensus.cc:686] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:11:51.481146 29650 consensus_queue.cc:237] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:51.484588 27617 catalog_manager.cc:5484] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:51.603615 29798 raft_consensus.cc:482] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:51.603808 29798 raft_consensus.cc:504] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:51.604578 29798 leader_election.cc:288] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:11:51.605120 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9820951bffc749139229e2b79c8ab597" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:51.605137 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9820951bffc749139229e2b79c8ab597" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:11:51.605351 28022 raft_consensus.cc:2454] T 9820951bffc749139229e2b79c8ab597 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:11:51.605355 27761 raft_consensus.cc:2454] T 9820951bffc749139229e2b79c8ab597 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:11:51.605741 27827 leader_election.cc:302] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aad81047009a4ffcbc450b60030583b5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:51.606016 29798 raft_consensus.cc:2781] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:51.606137 29798 raft_consensus.cc:482] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:51.606226 29798 raft_consensus.cc:3037] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:51.608306 29798 raft_consensus.cc:504] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:51.608846 29798 leader_election.cc:288] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Requested vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:11:51.609362 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9820951bffc749139229e2b79c8ab597" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:11:51.609403 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9820951bffc749139229e2b79c8ab597" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:51.609555 27761 raft_consensus.cc:3037] T 9820951bffc749139229e2b79c8ab597 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:51.609659 28022 raft_consensus.cc:3037] T 9820951bffc749139229e2b79c8ab597 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:51.611605 27761 raft_consensus.cc:2454] T 9820951bffc749139229e2b79c8ab597 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:11:51.612035 27827 leader_election.cc:302] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:51.612325 29798 raft_consensus.cc:2781] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:51.612757 29798 raft_consensus.cc:686] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Becoming Leader. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Running, Role: LEADER
I20240501 08:11:51.613107 29798 consensus_queue.cc:237] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:51.613466 28022 raft_consensus.cc:2454] T 9820951bffc749139229e2b79c8ab597 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:11:51.616823 27617 catalog_manager.cc:5484] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af reported cstate change: term changed from 0 to 1, leader changed from <none> to f1525cbc82a24d13b65f136695c903af (127.21.84.130). New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:51.722649 29650 raft_consensus.cc:482] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:51.722833 29650 raft_consensus.cc:504] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:51.723480 29650 leader_election.cc:288] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:11:51.723918 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb7dae019c7e4842b0daa35ee4b40367" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:11:51.723934 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb7dae019c7e4842b0daa35ee4b40367" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:51.724146 28150 raft_consensus.cc:2454] T bb7dae019c7e4842b0daa35ee4b40367 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:11:51.724170 28022 raft_consensus.cc:2454] T bb7dae019c7e4842b0daa35ee4b40367 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:11:51.724550 27694 leader_election.cc:302] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:11:51.724849 29650 raft_consensus.cc:2781] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:51.724967 29650 raft_consensus.cc:482] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:51.725069 29650 raft_consensus.cc:3037] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:51.727273 29650 raft_consensus.cc:504] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:51.727916 29650 leader_election.cc:288] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Requested vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:11:51.728209 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb7dae019c7e4842b0daa35ee4b40367" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:11:51.728364 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb7dae019c7e4842b0daa35ee4b40367" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:51.728432 28150 raft_consensus.cc:3037] T bb7dae019c7e4842b0daa35ee4b40367 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:51.728557 28022 raft_consensus.cc:3037] T bb7dae019c7e4842b0daa35ee4b40367 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:51.731324 28150 raft_consensus.cc:2454] T bb7dae019c7e4842b0daa35ee4b40367 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:11:51.731336 28022 raft_consensus.cc:2454] T bb7dae019c7e4842b0daa35ee4b40367 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:11:51.731750 27694 leader_election.cc:302] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:11:51.732126 29650 raft_consensus.cc:2781] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:51.732263 29650 raft_consensus.cc:686] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:11:51.732486 29650 consensus_queue.cc:237] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:51.736624 27617 catalog_manager.cc:5484] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:51.913461 29650 consensus_queue.cc:1035] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:51.926625 29650 consensus_queue.cc:1035] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:52.006909 29798 consensus_queue.cc:1035] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:52.016062 29798 consensus_queue.cc:1035] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:52.152181 29871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:52.152554 29871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 08:11:52.155062 29876 consensus_queue.cc:1035] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:52.160081 29876 consensus_queue.cc:1035] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:52.163828 29871 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_13'
INSERT report
rows total: 2000
time total: 235.209 ms
time per row: 0.117605 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 08:11:53.527436 27612 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalance_test_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:53.537130 27739 tablet_service.cc:1467] Processing CreateTablet for tablet 60d21fc9d346438fb4ff01142fd3d255 (DEFAULT_TABLE table=rebalance_test_table_14 [id=eb27ffc4ce874520a9b72904cae223b2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:53.537635 27739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60d21fc9d346438fb4ff01142fd3d255. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:53.537729 27871 tablet_service.cc:1467] Processing CreateTablet for tablet eadf85174d8746c691e825949dbbd678 (DEFAULT_TABLE table=rebalance_test_table_14 [id=eb27ffc4ce874520a9b72904cae223b2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:53.537837 28132 tablet_service.cc:1467] Processing CreateTablet for tablet eadf85174d8746c691e825949dbbd678 (DEFAULT_TABLE table=rebalance_test_table_14 [id=eb27ffc4ce874520a9b72904cae223b2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:53.538122 27871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eadf85174d8746c691e825949dbbd678. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:53.538260 28132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eadf85174d8746c691e825949dbbd678. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:53.537849 27872 tablet_service.cc:1467] Processing CreateTablet for tablet 1b2a23cb9a574422ac3fd3a0d9bcb782 (DEFAULT_TABLE table=rebalance_test_table_14 [id=eb27ffc4ce874520a9b72904cae223b2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:53.542474 28131 tablet_service.cc:1467] Processing CreateTablet for tablet 60d21fc9d346438fb4ff01142fd3d255 (DEFAULT_TABLE table=rebalance_test_table_14 [id=eb27ffc4ce874520a9b72904cae223b2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:53.542665 27872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b2a23cb9a574422ac3fd3a0d9bcb782. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:53.542850 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60d21fc9d346438fb4ff01142fd3d255. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:53.545647 28130 tablet_service.cc:1467] Processing CreateTablet for tablet 1b2a23cb9a574422ac3fd3a0d9bcb782 (DEFAULT_TABLE table=rebalance_test_table_14 [id=eb27ffc4ce874520a9b72904cae223b2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:53.546569 27741 tablet_service.cc:1467] Processing CreateTablet for tablet 1b2a23cb9a574422ac3fd3a0d9bcb782 (DEFAULT_TABLE table=rebalance_test_table_14 [id=eb27ffc4ce874520a9b72904cae223b2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:53.546588 27870 tablet_service.cc:1467] Processing CreateTablet for tablet 60d21fc9d346438fb4ff01142fd3d255 (DEFAULT_TABLE table=rebalance_test_table_14 [id=eb27ffc4ce874520a9b72904cae223b2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:53.546612 28002 tablet_service.cc:1467] Processing CreateTablet for tablet eadf85174d8746c691e825949dbbd678 (DEFAULT_TABLE table=rebalance_test_table_14 [id=eb27ffc4ce874520a9b72904cae223b2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:53.546978 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b2a23cb9a574422ac3fd3a0d9bcb782. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:53.546970 27870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60d21fc9d346438fb4ff01142fd3d255. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:53.547046 28002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eadf85174d8746c691e825949dbbd678. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:53.549343 29943 tablet_bootstrap.cc:492] T 60d21fc9d346438fb4ff01142fd3d255 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:53.550972 28130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b2a23cb9a574422ac3fd3a0d9bcb782. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:53.553251 29943 tablet_bootstrap.cc:654] T 60d21fc9d346438fb4ff01142fd3d255 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:53.562263 29947 tablet_bootstrap.cc:492] T eadf85174d8746c691e825949dbbd678 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:53.563941 29948 tablet_bootstrap.cc:492] T eadf85174d8746c691e825949dbbd678 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:53.566856 29948 tablet_bootstrap.cc:654] T eadf85174d8746c691e825949dbbd678 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:53.567574 29947 tablet_bootstrap.cc:654] T eadf85174d8746c691e825949dbbd678 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:53.573051 29947 tablet_bootstrap.cc:492] T eadf85174d8746c691e825949dbbd678 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:53.574206 29947 ts_tablet_manager.cc:1367] T eadf85174d8746c691e825949dbbd678 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.002s
I20240501 08:11:53.575135 29947 raft_consensus.cc:348] T eadf85174d8746c691e825949dbbd678 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:53.575505 29947 raft_consensus.cc:374] T eadf85174d8746c691e825949dbbd678 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:53.575627 29947 raft_consensus.cc:729] T eadf85174d8746c691e825949dbbd678 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:53.575923 29947 consensus_queue.cc:260] T eadf85174d8746c691e825949dbbd678 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:53.577605 29947 ts_tablet_manager.cc:1398] T eadf85174d8746c691e825949dbbd678 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 08:11:53.579988 29944 tablet_bootstrap.cc:492] T eadf85174d8746c691e825949dbbd678 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:53.583055 29944 tablet_bootstrap.cc:654] T eadf85174d8746c691e825949dbbd678 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:53.586429 29948 tablet_bootstrap.cc:492] T eadf85174d8746c691e825949dbbd678 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:53.586747 29948 ts_tablet_manager.cc:1367] T eadf85174d8746c691e825949dbbd678 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.000s
I20240501 08:11:53.587843 29948 raft_consensus.cc:348] T eadf85174d8746c691e825949dbbd678 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:53.588140 29948 raft_consensus.cc:374] T eadf85174d8746c691e825949dbbd678 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:53.588241 29948 raft_consensus.cc:729] T eadf85174d8746c691e825949dbbd678 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:53.588548 29948 consensus_queue.cc:260] T eadf85174d8746c691e825949dbbd678 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:53.589360 29948 ts_tablet_manager.cc:1398] T eadf85174d8746c691e825949dbbd678 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:53.589764 29948 tablet_bootstrap.cc:492] T 60d21fc9d346438fb4ff01142fd3d255 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:53.592643 29948 tablet_bootstrap.cc:654] T 60d21fc9d346438fb4ff01142fd3d255 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:53.593951 29944 tablet_bootstrap.cc:492] T eadf85174d8746c691e825949dbbd678 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:53.594298 29944 ts_tablet_manager.cc:1367] T eadf85174d8746c691e825949dbbd678 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.004s
I20240501 08:11:53.595718 29944 raft_consensus.cc:348] T eadf85174d8746c691e825949dbbd678 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:53.596087 29944 raft_consensus.cc:374] T eadf85174d8746c691e825949dbbd678 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:53.596243 29944 raft_consensus.cc:729] T eadf85174d8746c691e825949dbbd678 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:53.596788 29944 consensus_queue.cc:260] T eadf85174d8746c691e825949dbbd678 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:53.597831 29944 ts_tablet_manager.cc:1398] T eadf85174d8746c691e825949dbbd678 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:11:53.598162 29944 tablet_bootstrap.cc:492] T 60d21fc9d346438fb4ff01142fd3d255 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:53.600986 29944 tablet_bootstrap.cc:654] T 60d21fc9d346438fb4ff01142fd3d255 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:53.605958 29868 raft_consensus.cc:482] T eadf85174d8746c691e825949dbbd678 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:53.606119 29868 raft_consensus.cc:504] T eadf85174d8746c691e825949dbbd678 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:53.606958 29868 leader_election.cc:288] T eadf85174d8746c691e825949dbbd678 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:11:53.607698 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadf85174d8746c691e825949dbbd678" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:11:53.607944 28151 raft_consensus.cc:2454] T eadf85174d8746c691e825949dbbd678 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:11:53.608733 28019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadf85174d8746c691e825949dbbd678" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:53.608956 28019 raft_consensus.cc:2454] T eadf85174d8746c691e825949dbbd678 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:11:53.609355 27825 leader_election.cc:302] T eadf85174d8746c691e825949dbbd678 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:53.609719 29868 raft_consensus.cc:2781] T eadf85174d8746c691e825949dbbd678 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:53.609819 29868 raft_consensus.cc:482] T eadf85174d8746c691e825949dbbd678 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:53.609894 29868 raft_consensus.cc:3037] T eadf85174d8746c691e825949dbbd678 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:53.615545 29944 tablet_bootstrap.cc:492] T 60d21fc9d346438fb4ff01142fd3d255 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:53.615610 29868 raft_consensus.cc:504] T eadf85174d8746c691e825949dbbd678 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:53.615934 29944 ts_tablet_manager.cc:1367] T 60d21fc9d346438fb4ff01142fd3d255 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.001s
I20240501 08:11:53.617040 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadf85174d8746c691e825949dbbd678" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:11:53.617275 28151 raft_consensus.cc:3037] T eadf85174d8746c691e825949dbbd678 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:53.617456 29944 raft_consensus.cc:348] T 60d21fc9d346438fb4ff01142fd3d255 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:53.617766 29944 raft_consensus.cc:374] T 60d21fc9d346438fb4ff01142fd3d255 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:53.617852 29944 raft_consensus.cc:729] T 60d21fc9d346438fb4ff01142fd3d255 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:53.619232 29948 tablet_bootstrap.cc:492] T 60d21fc9d346438fb4ff01142fd3d255 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:53.619374 29948 ts_tablet_manager.cc:1367] T 60d21fc9d346438fb4ff01142fd3d255 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.004s
I20240501 08:11:53.620123 29943 tablet_bootstrap.cc:492] T 60d21fc9d346438fb4ff01142fd3d255 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:53.620352 28151 raft_consensus.cc:2454] T eadf85174d8746c691e825949dbbd678 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:11:53.620424 29943 ts_tablet_manager.cc:1367] T 60d21fc9d346438fb4ff01142fd3d255 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.071s user 0.019s sys 0.042s
I20240501 08:11:53.621021 27825 leader_election.cc:302] T eadf85174d8746c691e825949dbbd678 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:53.621430 29945 raft_consensus.cc:2781] T eadf85174d8746c691e825949dbbd678 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:53.621325 29943 raft_consensus.cc:348] T 60d21fc9d346438fb4ff01142fd3d255 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:53.621528 29943 raft_consensus.cc:374] T 60d21fc9d346438fb4ff01142fd3d255 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:53.621611 29943 raft_consensus.cc:729] T 60d21fc9d346438fb4ff01142fd3d255 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:53.621887 29943 consensus_queue.cc:260] T 60d21fc9d346438fb4ff01142fd3d255 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:53.622735 29943 ts_tablet_manager.cc:1398] T 60d21fc9d346438fb4ff01142fd3d255 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:53.623317 29943 tablet_bootstrap.cc:492] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:53.625685 29943 tablet_bootstrap.cc:654] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:53.625844 29868 leader_election.cc:288] T eadf85174d8746c691e825949dbbd678 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Requested vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:11:53.620227 29948 raft_consensus.cc:348] T 60d21fc9d346438fb4ff01142fd3d255 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:53.627512 29948 raft_consensus.cc:374] T 60d21fc9d346438fb4ff01142fd3d255 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:53.627844 29948 raft_consensus.cc:729] T 60d21fc9d346438fb4ff01142fd3d255 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:53.628492 29948 consensus_queue.cc:260] T 60d21fc9d346438fb4ff01142fd3d255 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:53.629868 29948 ts_tablet_manager.cc:1398] T 60d21fc9d346438fb4ff01142fd3d255 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.010s user 0.002s sys 0.001s
I20240501 08:11:53.634368 29943 tablet_bootstrap.cc:492] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:53.634627 29943 ts_tablet_manager.cc:1367] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20240501 08:11:53.635535 29943 raft_consensus.cc:348] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:53.635862 29943 raft_consensus.cc:374] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:53.636014 29943 raft_consensus.cc:729] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:53.636377 29943 consensus_queue.cc:260] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:53.636821 29948 tablet_bootstrap.cc:492] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:53.637192 29943 ts_tablet_manager.cc:1398] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:53.637233 29944 consensus_queue.cc:260] T 60d21fc9d346438fb4ff01142fd3d255 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:53.639077 29948 tablet_bootstrap.cc:654] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:53.640240 29944 ts_tablet_manager.cc:1398] T 60d21fc9d346438fb4ff01142fd3d255 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20240501 08:11:53.640967 29944 tablet_bootstrap.cc:492] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:53.642765 29944 tablet_bootstrap.cc:654] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:53.643163 29945 raft_consensus.cc:686] T eadf85174d8746c691e825949dbbd678 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Becoming Leader. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Running, Role: LEADER
I20240501 08:11:53.643527 29945 consensus_queue.cc:237] T eadf85174d8746c691e825949dbbd678 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:53.646241 28019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadf85174d8746c691e825949dbbd678" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:53.646400 29944 tablet_bootstrap.cc:492] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:53.646487 28019 raft_consensus.cc:3037] T eadf85174d8746c691e825949dbbd678 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:53.646530 29944 ts_tablet_manager.cc:1367] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240501 08:11:53.647115 29944 raft_consensus.cc:348] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:53.647264 29944 raft_consensus.cc:374] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:53.647346 29944 raft_consensus.cc:729] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:53.647528 29944 consensus_queue.cc:260] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:53.647972 29944 ts_tablet_manager.cc:1398] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:11:53.648691 28019 raft_consensus.cc:2454] T eadf85174d8746c691e825949dbbd678 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:11:53.650048 29948 tablet_bootstrap.cc:492] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:53.650230 29948 ts_tablet_manager.cc:1367] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240501 08:11:53.650023 27612 catalog_manager.cc:5484] T eadf85174d8746c691e825949dbbd678 P f1525cbc82a24d13b65f136695c903af reported cstate change: term changed from 0 to 1, leader changed from <none> to f1525cbc82a24d13b65f136695c903af (127.21.84.130). New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:53.651050 29948 raft_consensus.cc:348] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:53.651235 29948 raft_consensus.cc:374] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:53.651324 29948 raft_consensus.cc:729] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:53.651533 29948 consensus_queue.cc:260] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:53.652014 29948 ts_tablet_manager.cc:1398] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:11:53.814761 29872 raft_consensus.cc:482] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:53.814929 29872 raft_consensus.cc:504] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:53.815676 29872 leader_election.cc:288] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:53.816126 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b2a23cb9a574422ac3fd3a0d9bcb782" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:11:53.816222 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b2a23cb9a574422ac3fd3a0d9bcb782" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:11:53.816375 27890 raft_consensus.cc:2454] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:53.816473 27761 raft_consensus.cc:2454] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:53.816798 28088 leader_election.cc:302] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:53.817052 29872 raft_consensus.cc:2781] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:53.817147 29872 raft_consensus.cc:482] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:53.817234 29872 raft_consensus.cc:3037] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:53.819185 29872 raft_consensus.cc:504] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:53.819700 29872 leader_election.cc:288] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:53.820216 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b2a23cb9a574422ac3fd3a0d9bcb782" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:11:53.820223 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b2a23cb9a574422ac3fd3a0d9bcb782" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:11:53.820453 27890 raft_consensus.cc:3037] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:53.820448 27761 raft_consensus.cc:3037] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:53.823240 27761 raft_consensus.cc:2454] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:53.823240 27890 raft_consensus.cc:2454] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:53.823725 28087 leader_election.cc:302] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:11:53.824023 29872 raft_consensus.cc:2781] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:53.824152 29872 raft_consensus.cc:686] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:11:53.824400 29872 consensus_queue.cc:237] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:53.828027 27616 catalog_manager.cc:5484] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:54.012419 29872 raft_consensus.cc:482] T 60d21fc9d346438fb4ff01142fd3d255 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:54.012660 29872 raft_consensus.cc:504] T 60d21fc9d346438fb4ff01142fd3d255 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:54.013345 29872 leader_election.cc:288] T 60d21fc9d346438fb4ff01142fd3d255 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:54.014035 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d21fc9d346438fb4ff01142fd3d255" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:11:54.014035 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d21fc9d346438fb4ff01142fd3d255" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:11:54.014273 27890 raft_consensus.cc:2454] T 60d21fc9d346438fb4ff01142fd3d255 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:54.014284 27761 raft_consensus.cc:2454] T 60d21fc9d346438fb4ff01142fd3d255 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:54.014782 28088 leader_election.cc:302] T 60d21fc9d346438fb4ff01142fd3d255 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:54.015136 29872 raft_consensus.cc:2781] T 60d21fc9d346438fb4ff01142fd3d255 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:54.015223 29872 raft_consensus.cc:482] T 60d21fc9d346438fb4ff01142fd3d255 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:54.015340 29872 raft_consensus.cc:3037] T 60d21fc9d346438fb4ff01142fd3d255 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:54.017185 29872 raft_consensus.cc:504] T 60d21fc9d346438fb4ff01142fd3d255 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:54.017756 29872 leader_election.cc:288] T 60d21fc9d346438fb4ff01142fd3d255 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:54.018332 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d21fc9d346438fb4ff01142fd3d255" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:11:54.018334 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60d21fc9d346438fb4ff01142fd3d255" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:11:54.018515 27761 raft_consensus.cc:3037] T 60d21fc9d346438fb4ff01142fd3d255 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:54.018528 27890 raft_consensus.cc:3037] T 60d21fc9d346438fb4ff01142fd3d255 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:54.020466 27890 raft_consensus.cc:2454] T 60d21fc9d346438fb4ff01142fd3d255 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:54.020465 27761 raft_consensus.cc:2454] T 60d21fc9d346438fb4ff01142fd3d255 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:54.020954 28088 leader_election.cc:302] T 60d21fc9d346438fb4ff01142fd3d255 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:54.021267 29872 raft_consensus.cc:2781] T 60d21fc9d346438fb4ff01142fd3d255 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:54.021427 29872 raft_consensus.cc:686] T 60d21fc9d346438fb4ff01142fd3d255 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:11:54.021706 29872 consensus_queue.cc:237] T 60d21fc9d346438fb4ff01142fd3d255 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:54.025120 27616 catalog_manager.cc:5484] T 60d21fc9d346438fb4ff01142fd3d255 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:54.172093 29868 consensus_queue.cc:1035] T eadf85174d8746c691e825949dbbd678 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:54.179021 29965 consensus_queue.cc:1035] T eadf85174d8746c691e825949dbbd678 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:54.348460 29872 consensus_queue.cc:1035] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:54.352860 29872 consensus_queue.cc:1035] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:54.409229 29963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:54.409596 29963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:54.417819 29963 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_14'
I20240501 08:11:54.504727 29971 consensus_queue.cc:1035] T 60d21fc9d346438fb4ff01142fd3d255 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:54.512293 29872 consensus_queue.cc:1035] T 60d21fc9d346438fb4ff01142fd3d255 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 300.537 ms
time per row: 0.150269 ms
I20240501 08:11:54.870261 27619 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalance_test_table_15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:54.879566 28002 tablet_service.cc:1467] Processing CreateTablet for tablet 32a54b70dcc447828ec293a9ff0afb80 (DEFAULT_TABLE table=rebalance_test_table_15 [id=d3eb6ca7d8cf4bfdb114edb23abb2ed2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:54.880043 28002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a54b70dcc447828ec293a9ff0afb80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:54.880194 28000 tablet_service.cc:1467] Processing CreateTablet for tablet a05223cb2f20478b8d29ffa065b4c67b (DEFAULT_TABLE table=rebalance_test_table_15 [id=d3eb6ca7d8cf4bfdb114edb23abb2ed2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:54.880517 28000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a05223cb2f20478b8d29ffa065b4c67b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:54.881541 27872 tablet_service.cc:1467] Processing CreateTablet for tablet 5536362915034f3eb86138b45d71b8ad (DEFAULT_TABLE table=rebalance_test_table_15 [id=d3eb6ca7d8cf4bfdb114edb23abb2ed2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:54.882006 27872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5536362915034f3eb86138b45d71b8ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:54.884053 27871 tablet_service.cc:1467] Processing CreateTablet for tablet 32a54b70dcc447828ec293a9ff0afb80 (DEFAULT_TABLE table=rebalance_test_table_15 [id=d3eb6ca7d8cf4bfdb114edb23abb2ed2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:54.884434 27871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a54b70dcc447828ec293a9ff0afb80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:54.886384 27739 tablet_service.cc:1467] Processing CreateTablet for tablet a05223cb2f20478b8d29ffa065b4c67b (DEFAULT_TABLE table=rebalance_test_table_15 [id=d3eb6ca7d8cf4bfdb114edb23abb2ed2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:54.886806 27739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a05223cb2f20478b8d29ffa065b4c67b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:54.887822 27741 tablet_service.cc:1467] Processing CreateTablet for tablet 32a54b70dcc447828ec293a9ff0afb80 (DEFAULT_TABLE table=rebalance_test_table_15 [id=d3eb6ca7d8cf4bfdb114edb23abb2ed2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:54.888156 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a54b70dcc447828ec293a9ff0afb80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:54.891062 28130 tablet_service.cc:1467] Processing CreateTablet for tablet a05223cb2f20478b8d29ffa065b4c67b (DEFAULT_TABLE table=rebalance_test_table_15 [id=d3eb6ca7d8cf4bfdb114edb23abb2ed2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:54.891242 28001 tablet_service.cc:1467] Processing CreateTablet for tablet 5536362915034f3eb86138b45d71b8ad (DEFAULT_TABLE table=rebalance_test_table_15 [id=d3eb6ca7d8cf4bfdb114edb23abb2ed2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:54.891503 28130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a05223cb2f20478b8d29ffa065b4c67b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:54.891652 28001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5536362915034f3eb86138b45d71b8ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:54.895961 28132 tablet_service.cc:1467] Processing CreateTablet for tablet 5536362915034f3eb86138b45d71b8ad (DEFAULT_TABLE table=rebalance_test_table_15 [id=d3eb6ca7d8cf4bfdb114edb23abb2ed2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:54.896363 28132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5536362915034f3eb86138b45d71b8ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:54.898726 30031 tablet_bootstrap.cc:492] T 5536362915034f3eb86138b45d71b8ad P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:54.906119 30032 tablet_bootstrap.cc:492] T a05223cb2f20478b8d29ffa065b4c67b P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:54.907825 30033 tablet_bootstrap.cc:492] T 32a54b70dcc447828ec293a9ff0afb80 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:54.907856 30034 tablet_bootstrap.cc:492] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:54.908628 30031 tablet_bootstrap.cc:654] T 5536362915034f3eb86138b45d71b8ad P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:54.909667 30033 tablet_bootstrap.cc:654] T 32a54b70dcc447828ec293a9ff0afb80 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:54.909803 30034 tablet_bootstrap.cc:654] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:54.910599 30032 tablet_bootstrap.cc:654] T a05223cb2f20478b8d29ffa065b4c67b P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:54.913949 30034 tablet_bootstrap.cc:492] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:54.914206 30034 ts_tablet_manager.cc:1367] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 08:11:54.914247 30031 tablet_bootstrap.cc:492] T 5536362915034f3eb86138b45d71b8ad P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:54.914455 30031 ts_tablet_manager.cc:1367] T 5536362915034f3eb86138b45d71b8ad P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20240501 08:11:54.914618 30032 tablet_bootstrap.cc:492] T a05223cb2f20478b8d29ffa065b4c67b P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:54.914769 30033 tablet_bootstrap.cc:492] T 32a54b70dcc447828ec293a9ff0afb80 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:54.914860 30032 ts_tablet_manager.cc:1367] T a05223cb2f20478b8d29ffa065b4c67b P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240501 08:11:54.914911 30033 ts_tablet_manager.cc:1367] T 32a54b70dcc447828ec293a9ff0afb80 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 08:11:54.915190 30034 raft_consensus.cc:348] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:54.915318 30031 raft_consensus.cc:348] T 5536362915034f3eb86138b45d71b8ad P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:54.915467 30034 raft_consensus.cc:374] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:54.915567 30031 raft_consensus.cc:374] T 5536362915034f3eb86138b45d71b8ad P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:54.915632 30034 raft_consensus.cc:729] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:54.915700 30031 raft_consensus.cc:729] T 5536362915034f3eb86138b45d71b8ad P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:54.915673 30033 raft_consensus.cc:348] T 32a54b70dcc447828ec293a9ff0afb80 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:54.915714 30032 raft_consensus.cc:348] T a05223cb2f20478b8d29ffa065b4c67b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:54.915925 30032 raft_consensus.cc:374] T a05223cb2f20478b8d29ffa065b4c67b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:54.916002 30033 raft_consensus.cc:374] T 32a54b70dcc447828ec293a9ff0afb80 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:54.915966 30031 consensus_queue.cc:260] T 5536362915034f3eb86138b45d71b8ad P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:54.915985 30034 consensus_queue.cc:260] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:54.916108 30033 raft_consensus.cc:729] T 32a54b70dcc447828ec293a9ff0afb80 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:54.916062 30032 raft_consensus.cc:729] T a05223cb2f20478b8d29ffa065b4c67b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:54.916400 30032 consensus_queue.cc:260] T a05223cb2f20478b8d29ffa065b4c67b P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:54.916985 30032 ts_tablet_manager.cc:1398] T a05223cb2f20478b8d29ffa065b4c67b P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:11:54.917552 30034 ts_tablet_manager.cc:1398] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:54.918198 30032 tablet_bootstrap.cc:492] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:54.918406 30031 ts_tablet_manager.cc:1398] T 5536362915034f3eb86138b45d71b8ad P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:11:54.918807 30031 tablet_bootstrap.cc:492] T 32a54b70dcc447828ec293a9ff0afb80 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:54.920004 30032 tablet_bootstrap.cc:654] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:54.919353 30033 consensus_queue.cc:260] T 32a54b70dcc447828ec293a9ff0afb80 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:54.921458 30033 ts_tablet_manager.cc:1398] T 32a54b70dcc447828ec293a9ff0afb80 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:11:54.922499 30033 tablet_bootstrap.cc:492] T a05223cb2f20478b8d29ffa065b4c67b P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:54.922735 30034 tablet_bootstrap.cc:492] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:54.924495 30034 tablet_bootstrap.cc:654] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:54.926541 30033 tablet_bootstrap.cc:654] T a05223cb2f20478b8d29ffa065b4c67b P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:54.921072 30031 tablet_bootstrap.cc:654] T 32a54b70dcc447828ec293a9ff0afb80 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:54.931753 30032 tablet_bootstrap.cc:492] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:54.932026 30032 ts_tablet_manager.cc:1367] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.008s
I20240501 08:11:54.932902 30032 raft_consensus.cc:348] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:54.933321 30032 raft_consensus.cc:374] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:54.933455 30032 raft_consensus.cc:729] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:54.933763 30032 consensus_queue.cc:260] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:54.935482 30032 ts_tablet_manager.cc:1398] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:11:54.936128 30031 tablet_bootstrap.cc:492] T 32a54b70dcc447828ec293a9ff0afb80 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:54.936375 30034 tablet_bootstrap.cc:492] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:54.936378 30031 ts_tablet_manager.cc:1367] T 32a54b70dcc447828ec293a9ff0afb80 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240501 08:11:54.936582 30034 ts_tablet_manager.cc:1367] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.007s
I20240501 08:11:54.936955 30033 tablet_bootstrap.cc:492] T a05223cb2f20478b8d29ffa065b4c67b P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:54.937283 30033 ts_tablet_manager.cc:1367] T a05223cb2f20478b8d29ffa065b4c67b P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20240501 08:11:54.937213 30031 raft_consensus.cc:348] T 32a54b70dcc447828ec293a9ff0afb80 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:54.937429 30034 raft_consensus.cc:348] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:54.937655 30034 raft_consensus.cc:374] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:54.937741 30034 raft_consensus.cc:729] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:54.937906 30031 raft_consensus.cc:374] T 32a54b70dcc447828ec293a9ff0afb80 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:54.938055 30031 raft_consensus.cc:729] T 32a54b70dcc447828ec293a9ff0afb80 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:54.938187 30033 raft_consensus.cc:348] T a05223cb2f20478b8d29ffa065b4c67b P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:54.938459 30033 raft_consensus.cc:374] T a05223cb2f20478b8d29ffa065b4c67b P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:54.938359 30031 consensus_queue.cc:260] T 32a54b70dcc447828ec293a9ff0afb80 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:54.938599 30033 raft_consensus.cc:729] T a05223cb2f20478b8d29ffa065b4c67b P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:54.938925 30033 consensus_queue.cc:260] T a05223cb2f20478b8d29ffa065b4c67b P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:54.939079 30034 consensus_queue.cc:260] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:54.939702 30034 ts_tablet_manager.cc:1398] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 08:11:54.940984 30031 ts_tablet_manager.cc:1398] T 32a54b70dcc447828ec293a9ff0afb80 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 08:11:54.941223 30033 ts_tablet_manager.cc:1398] T a05223cb2f20478b8d29ffa065b4c67b P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:11:54.941571 30033 tablet_bootstrap.cc:492] T 5536362915034f3eb86138b45d71b8ad P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:54.944977 30033 tablet_bootstrap.cc:654] T 5536362915034f3eb86138b45d71b8ad P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:54.948993 30033 tablet_bootstrap.cc:492] T 5536362915034f3eb86138b45d71b8ad P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:54.949143 30033 ts_tablet_manager.cc:1367] T 5536362915034f3eb86138b45d71b8ad P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240501 08:11:54.949756 30033 raft_consensus.cc:348] T 5536362915034f3eb86138b45d71b8ad P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:54.949918 30033 raft_consensus.cc:374] T 5536362915034f3eb86138b45d71b8ad P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:54.950049 30033 raft_consensus.cc:729] T 5536362915034f3eb86138b45d71b8ad P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:54.950240 30033 consensus_queue.cc:260] T 5536362915034f3eb86138b45d71b8ad P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:54.950666 30033 ts_tablet_manager.cc:1398] T 5536362915034f3eb86138b45d71b8ad P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 08:11:54.975574 29936 raft_consensus.cc:482] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:54.975754 29936 raft_consensus.cc:504] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:54.976528 29936 leader_election.cc:288] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:11:54.977021 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a54b70dcc447828ec293a9ff0afb80" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:54.977084 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a54b70dcc447828ec293a9ff0afb80" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:11:54.977279 28022 raft_consensus.cc:2454] T 32a54b70dcc447828ec293a9ff0afb80 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:11:54.977363 27890 raft_consensus.cc:2454] T 32a54b70dcc447828ec293a9ff0afb80 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:11:54.977715 27696 leader_election.cc:302] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:54.978020 29936 raft_consensus.cc:2781] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:54.978130 29936 raft_consensus.cc:482] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:54.978246 29936 raft_consensus.cc:3037] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:54.980195 29936 raft_consensus.cc:504] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:54.980746 29936 leader_election.cc:288] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Requested vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:11:54.981283 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a54b70dcc447828ec293a9ff0afb80" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:54.981298 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a54b70dcc447828ec293a9ff0afb80" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:11:54.981529 28022 raft_consensus.cc:3037] T 32a54b70dcc447828ec293a9ff0afb80 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:54.981528 27890 raft_consensus.cc:3037] T 32a54b70dcc447828ec293a9ff0afb80 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:54.984385 28022 raft_consensus.cc:2454] T 32a54b70dcc447828ec293a9ff0afb80 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:11:54.984382 27890 raft_consensus.cc:2454] T 32a54b70dcc447828ec293a9ff0afb80 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:11:54.984863 27696 leader_election.cc:302] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:54.985168 29650 raft_consensus.cc:2781] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:54.985313 29650 raft_consensus.cc:686] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:11:54.985555 29650 consensus_queue.cc:237] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:54.988886 27611 catalog_manager.cc:5484] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:55.014429 29971 raft_consensus.cc:482] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:55.014626 29971 raft_consensus.cc:504] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:55.015244 29971 leader_election.cc:288] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:55.015713 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05223cb2f20478b8d29ffa065b4c67b" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:55.015807 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05223cb2f20478b8d29ffa065b4c67b" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:11:55.015949 28022 raft_consensus.cc:2454] T a05223cb2f20478b8d29ffa065b4c67b P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:55.016037 27761 raft_consensus.cc:2454] T a05223cb2f20478b8d29ffa065b4c67b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:55.016398 28087 leader_election.cc:302] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:55.016712 29971 raft_consensus.cc:2781] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:55.016832 29971 raft_consensus.cc:482] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:55.016955 29971 raft_consensus.cc:3037] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:55.019086 29971 raft_consensus.cc:504] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:55.019714 29971 leader_election.cc:288] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:55.020202 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05223cb2f20478b8d29ffa065b4c67b" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:55.020519 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05223cb2f20478b8d29ffa065b4c67b" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:11:55.020641 28022 raft_consensus.cc:3037] T a05223cb2f20478b8d29ffa065b4c67b P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:55.020740 27761 raft_consensus.cc:3037] T a05223cb2f20478b8d29ffa065b4c67b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:55.023268 27761 raft_consensus.cc:2454] T a05223cb2f20478b8d29ffa065b4c67b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:55.023636 28022 raft_consensus.cc:2454] T a05223cb2f20478b8d29ffa065b4c67b P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:55.023716 28087 leader_election.cc:302] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:11:55.024017 29971 raft_consensus.cc:2781] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:55.024205 29971 raft_consensus.cc:686] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:11:55.024868 29971 consensus_queue.cc:237] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:55.028317 27611 catalog_manager.cc:5484] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:55.038564 29971 raft_consensus.cc:482] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:55.038771 29971 raft_consensus.cc:504] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:55.039420 29971 leader_election.cc:288] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:11:55.039887 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5536362915034f3eb86138b45d71b8ad" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:55.039939 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5536362915034f3eb86138b45d71b8ad" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:11:55.040127 28021 raft_consensus.cc:2454] T 5536362915034f3eb86138b45d71b8ad P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:55.040194 27890 raft_consensus.cc:2454] T 5536362915034f3eb86138b45d71b8ad P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:55.040606 28087 leader_election.cc:302] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:55.040925 29971 raft_consensus.cc:2781] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:55.041054 29971 raft_consensus.cc:482] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:55.041142 29971 raft_consensus.cc:3037] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:55.043133 29971 raft_consensus.cc:504] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:55.043913 29971 leader_election.cc:288] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:11:55.044492 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5536362915034f3eb86138b45d71b8ad" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:55.044684 28021 raft_consensus.cc:3037] T 5536362915034f3eb86138b45d71b8ad P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:55.044739 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5536362915034f3eb86138b45d71b8ad" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:11:55.044924 27890 raft_consensus.cc:3037] T 5536362915034f3eb86138b45d71b8ad P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:55.046787 27890 raft_consensus.cc:2454] T 5536362915034f3eb86138b45d71b8ad P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:55.047286 28088 leader_election.cc:302] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:55.047480 28021 raft_consensus.cc:2454] T 5536362915034f3eb86138b45d71b8ad P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:55.047674 29971 raft_consensus.cc:2781] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:55.047843 29971 raft_consensus.cc:686] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:11:55.048151 29971 consensus_queue.cc:237] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:55.053004 27611 catalog_manager.cc:5484] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:55.365121 29936 consensus_queue.cc:1035] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:55.382982 29936 consensus_queue.cc:1035] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:55.423527 29971 consensus_queue.cc:1035] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:55.428879 29971 consensus_queue.cc:1035] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:55.479298 30048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:55.479739 30048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:55.487727 30048 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:11:55.515955 29971 consensus_queue.cc:1035] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_15'
I20240501 08:11:55.522215 30010 consensus_queue.cc:1035] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 271.856 ms
time per row: 0.135928 ms
I20240501 08:11:55.921556 27618 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalance_test_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:55.930783 28002 tablet_service.cc:1467] Processing CreateTablet for tablet 271d8e2e56d2468b8f0f9b7835c74a62 (DEFAULT_TABLE table=rebalance_test_table_16 [id=5e12abb0d0d6480c8906c3ec242309fc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:55.931102 27739 tablet_service.cc:1467] Processing CreateTablet for tablet 271d8e2e56d2468b8f0f9b7835c74a62 (DEFAULT_TABLE table=rebalance_test_table_16 [id=5e12abb0d0d6480c8906c3ec242309fc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:55.931118 28130 tablet_service.cc:1467] Processing CreateTablet for tablet 188e377fef614ce1801a8f9306d789e1 (DEFAULT_TABLE table=rebalance_test_table_16 [id=5e12abb0d0d6480c8906c3ec242309fc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:55.931311 28002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 271d8e2e56d2468b8f0f9b7835c74a62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:55.931458 28130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 188e377fef614ce1801a8f9306d789e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:55.931493 27739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 271d8e2e56d2468b8f0f9b7835c74a62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:55.934615 27872 tablet_service.cc:1467] Processing CreateTablet for tablet 271d8e2e56d2468b8f0f9b7835c74a62 (DEFAULT_TABLE table=rebalance_test_table_16 [id=5e12abb0d0d6480c8906c3ec242309fc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:55.935086 27872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 271d8e2e56d2468b8f0f9b7835c74a62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:55.935856 28132 tablet_service.cc:1467] Processing CreateTablet for tablet 428edf53f79e4b6d9f4b455cba16e644 (DEFAULT_TABLE table=rebalance_test_table_16 [id=5e12abb0d0d6480c8906c3ec242309fc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:55.936228 28132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 428edf53f79e4b6d9f4b455cba16e644. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:55.936219 27740 tablet_service.cc:1467] Processing CreateTablet for tablet 428edf53f79e4b6d9f4b455cba16e644 (DEFAULT_TABLE table=rebalance_test_table_16 [id=5e12abb0d0d6480c8906c3ec242309fc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:55.936542 27740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 428edf53f79e4b6d9f4b455cba16e644. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:55.938314 27741 tablet_service.cc:1467] Processing CreateTablet for tablet 188e377fef614ce1801a8f9306d789e1 (DEFAULT_TABLE table=rebalance_test_table_16 [id=5e12abb0d0d6480c8906c3ec242309fc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:55.938604 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 188e377fef614ce1801a8f9306d789e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:55.942806 28001 tablet_service.cc:1467] Processing CreateTablet for tablet 428edf53f79e4b6d9f4b455cba16e644 (DEFAULT_TABLE table=rebalance_test_table_16 [id=5e12abb0d0d6480c8906c3ec242309fc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:55.943248 28001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 428edf53f79e4b6d9f4b455cba16e644. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:55.944384 27871 tablet_service.cc:1467] Processing CreateTablet for tablet 188e377fef614ce1801a8f9306d789e1 (DEFAULT_TABLE table=rebalance_test_table_16 [id=5e12abb0d0d6480c8906c3ec242309fc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:55.944859 27871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 188e377fef614ce1801a8f9306d789e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:55.950131 30116 tablet_bootstrap.cc:492] T 271d8e2e56d2468b8f0f9b7835c74a62 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:55.953701 30116 tablet_bootstrap.cc:654] T 271d8e2e56d2468b8f0f9b7835c74a62 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:55.957221 30114 tablet_bootstrap.cc:492] T 428edf53f79e4b6d9f4b455cba16e644 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:55.960281 30114 tablet_bootstrap.cc:654] T 428edf53f79e4b6d9f4b455cba16e644 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:55.969816 30114 tablet_bootstrap.cc:492] T 428edf53f79e4b6d9f4b455cba16e644 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:55.970167 30114 ts_tablet_manager.cc:1367] T 428edf53f79e4b6d9f4b455cba16e644 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.008s
I20240501 08:11:55.970278 30118 tablet_bootstrap.cc:492] T 271d8e2e56d2468b8f0f9b7835c74a62 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:55.971144 30114 raft_consensus.cc:348] T 428edf53f79e4b6d9f4b455cba16e644 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:55.971518 30114 raft_consensus.cc:374] T 428edf53f79e4b6d9f4b455cba16e644 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:55.971665 30114 raft_consensus.cc:729] T 428edf53f79e4b6d9f4b455cba16e644 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:55.972029 30114 consensus_queue.cc:260] T 428edf53f79e4b6d9f4b455cba16e644 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:55.972191 30118 tablet_bootstrap.cc:654] T 271d8e2e56d2468b8f0f9b7835c74a62 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:55.972898 30114 ts_tablet_manager.cc:1398] T 428edf53f79e4b6d9f4b455cba16e644 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:11:55.973843 30115 tablet_bootstrap.cc:492] T 428edf53f79e4b6d9f4b455cba16e644 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:55.974956 30114 tablet_bootstrap.cc:492] T 188e377fef614ce1801a8f9306d789e1 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:55.976323 30115 tablet_bootstrap.cc:654] T 428edf53f79e4b6d9f4b455cba16e644 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:55.977509 30114 tablet_bootstrap.cc:654] T 188e377fef614ce1801a8f9306d789e1 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:55.979523 30118 tablet_bootstrap.cc:492] T 271d8e2e56d2468b8f0f9b7835c74a62 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:55.979698 30116 tablet_bootstrap.cc:492] T 271d8e2e56d2468b8f0f9b7835c74a62 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:55.979859 30116 ts_tablet_manager.cc:1367] T 271d8e2e56d2468b8f0f9b7835c74a62 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.007s
I20240501 08:11:55.980319 30115 tablet_bootstrap.cc:492] T 428edf53f79e4b6d9f4b455cba16e644 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:55.980489 30115 ts_tablet_manager.cc:1367] T 428edf53f79e4b6d9f4b455cba16e644 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20240501 08:11:55.980680 30116 raft_consensus.cc:348] T 271d8e2e56d2468b8f0f9b7835c74a62 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:55.980885 30116 raft_consensus.cc:374] T 271d8e2e56d2468b8f0f9b7835c74a62 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:55.981021 30116 raft_consensus.cc:729] T 271d8e2e56d2468b8f0f9b7835c74a62 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:55.981261 30116 consensus_queue.cc:260] T 271d8e2e56d2468b8f0f9b7835c74a62 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:55.981293 30115 raft_consensus.cc:348] T 428edf53f79e4b6d9f4b455cba16e644 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:55.981474 30115 raft_consensus.cc:374] T 428edf53f79e4b6d9f4b455cba16e644 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:55.981561 30115 raft_consensus.cc:729] T 428edf53f79e4b6d9f4b455cba16e644 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:55.981781 30115 consensus_queue.cc:260] T 428edf53f79e4b6d9f4b455cba16e644 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:55.982412 30115 ts_tablet_manager.cc:1398] T 428edf53f79e4b6d9f4b455cba16e644 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:11:55.982743 30115 tablet_bootstrap.cc:492] T 271d8e2e56d2468b8f0f9b7835c74a62 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:55.982878 30116 ts_tablet_manager.cc:1398] T 271d8e2e56d2468b8f0f9b7835c74a62 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:11:55.983196 30116 tablet_bootstrap.cc:492] T 428edf53f79e4b6d9f4b455cba16e644 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:55.984097 30115 tablet_bootstrap.cc:654] T 271d8e2e56d2468b8f0f9b7835c74a62 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:55.984688 30118 ts_tablet_manager.cc:1367] T 271d8e2e56d2468b8f0f9b7835c74a62 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.008s
I20240501 08:11:55.985677 30114 tablet_bootstrap.cc:492] T 188e377fef614ce1801a8f9306d789e1 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:55.985541 30118 raft_consensus.cc:348] T 271d8e2e56d2468b8f0f9b7835c74a62 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:55.985749 30116 tablet_bootstrap.cc:654] T 428edf53f79e4b6d9f4b455cba16e644 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:55.985853 30118 raft_consensus.cc:374] T 271d8e2e56d2468b8f0f9b7835c74a62 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:55.985858 30114 ts_tablet_manager.cc:1367] T 188e377fef614ce1801a8f9306d789e1 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.001s
I20240501 08:11:55.986336 30118 raft_consensus.cc:729] T 271d8e2e56d2468b8f0f9b7835c74a62 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:55.986647 30114 raft_consensus.cc:348] T 188e377fef614ce1801a8f9306d789e1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:55.986861 30114 raft_consensus.cc:374] T 188e377fef614ce1801a8f9306d789e1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:55.986637 30118 consensus_queue.cc:260] T 271d8e2e56d2468b8f0f9b7835c74a62 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:55.987464 30114 raft_consensus.cc:729] T 188e377fef614ce1801a8f9306d789e1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:55.987632 30118 ts_tablet_manager.cc:1398] T 271d8e2e56d2468b8f0f9b7835c74a62 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:11:55.987715 30114 consensus_queue.cc:260] T 188e377fef614ce1801a8f9306d789e1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:55.988248 30118 tablet_bootstrap.cc:492] T 188e377fef614ce1801a8f9306d789e1 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:55.988795 30114 ts_tablet_manager.cc:1398] T 188e377fef614ce1801a8f9306d789e1 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 08:11:55.990478 30118 tablet_bootstrap.cc:654] T 188e377fef614ce1801a8f9306d789e1 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:55.991497 30115 tablet_bootstrap.cc:492] T 271d8e2e56d2468b8f0f9b7835c74a62 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:55.991670 30115 ts_tablet_manager.cc:1367] T 271d8e2e56d2468b8f0f9b7835c74a62 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20240501 08:11:55.992455 30115 raft_consensus.cc:348] T 271d8e2e56d2468b8f0f9b7835c74a62 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:55.992692 30115 raft_consensus.cc:374] T 271d8e2e56d2468b8f0f9b7835c74a62 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:55.992779 30115 raft_consensus.cc:729] T 271d8e2e56d2468b8f0f9b7835c74a62 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:55.993034 30115 consensus_queue.cc:260] T 271d8e2e56d2468b8f0f9b7835c74a62 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:55.993547 30115 ts_tablet_manager.cc:1398] T 271d8e2e56d2468b8f0f9b7835c74a62 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:11:55.993814 30115 tablet_bootstrap.cc:492] T 188e377fef614ce1801a8f9306d789e1 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:55.994818 30118 tablet_bootstrap.cc:492] T 188e377fef614ce1801a8f9306d789e1 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:55.994972 30118 ts_tablet_manager.cc:1367] T 188e377fef614ce1801a8f9306d789e1 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20240501 08:11:55.995288 30115 tablet_bootstrap.cc:654] T 188e377fef614ce1801a8f9306d789e1 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:55.995692 30118 raft_consensus.cc:348] T 188e377fef614ce1801a8f9306d789e1 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:55.995918 30118 raft_consensus.cc:374] T 188e377fef614ce1801a8f9306d789e1 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:55.995993 30118 raft_consensus.cc:729] T 188e377fef614ce1801a8f9306d789e1 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:55.996203 30118 consensus_queue.cc:260] T 188e377fef614ce1801a8f9306d789e1 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:55.996883 30118 ts_tablet_manager.cc:1398] T 188e377fef614ce1801a8f9306d789e1 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:11:55.997637 30116 tablet_bootstrap.cc:492] T 428edf53f79e4b6d9f4b455cba16e644 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:55.997825 30116 ts_tablet_manager.cc:1367] T 428edf53f79e4b6d9f4b455cba16e644 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.009s
I20240501 08:11:55.999228 30116 raft_consensus.cc:348] T 428edf53f79e4b6d9f4b455cba16e644 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:55.999471 30116 raft_consensus.cc:374] T 428edf53f79e4b6d9f4b455cba16e644 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:55.999567 30116 raft_consensus.cc:729] T 428edf53f79e4b6d9f4b455cba16e644 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:55.999990 30116 consensus_queue.cc:260] T 428edf53f79e4b6d9f4b455cba16e644 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:56.000627 30116 ts_tablet_manager.cc:1398] T 428edf53f79e4b6d9f4b455cba16e644 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:56.003283 30115 tablet_bootstrap.cc:492] T 188e377fef614ce1801a8f9306d789e1 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:56.003443 30115 ts_tablet_manager.cc:1367] T 188e377fef614ce1801a8f9306d789e1 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240501 08:11:56.004235 30115 raft_consensus.cc:348] T 188e377fef614ce1801a8f9306d789e1 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:56.004436 30115 raft_consensus.cc:374] T 188e377fef614ce1801a8f9306d789e1 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:56.004531 30115 raft_consensus.cc:729] T 188e377fef614ce1801a8f9306d789e1 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:56.004760 30115 consensus_queue.cc:260] T 188e377fef614ce1801a8f9306d789e1 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:56.005342 30115 ts_tablet_manager.cc:1398] T 188e377fef614ce1801a8f9306d789e1 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:11:56.017735 29971 raft_consensus.cc:482] T 428edf53f79e4b6d9f4b455cba16e644 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:56.017956 29971 raft_consensus.cc:504] T 428edf53f79e4b6d9f4b455cba16e644 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:56.018644 29971 leader_election.cc:288] T 428edf53f79e4b6d9f4b455cba16e644 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:56.018973 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "428edf53f79e4b6d9f4b455cba16e644" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:56.019040 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "428edf53f79e4b6d9f4b455cba16e644" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:11:56.019186 28021 raft_consensus.cc:2454] T 428edf53f79e4b6d9f4b455cba16e644 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:56.019321 27759 raft_consensus.cc:2454] T 428edf53f79e4b6d9f4b455cba16e644 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:56.019594 28087 leader_election.cc:302] T 428edf53f79e4b6d9f4b455cba16e644 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:56.019827 29971 raft_consensus.cc:2781] T 428edf53f79e4b6d9f4b455cba16e644 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:56.019927 29971 raft_consensus.cc:482] T 428edf53f79e4b6d9f4b455cba16e644 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:56.020009 29971 raft_consensus.cc:3037] T 428edf53f79e4b6d9f4b455cba16e644 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:56.021859 29971 raft_consensus.cc:504] T 428edf53f79e4b6d9f4b455cba16e644 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:56.022437 29971 leader_election.cc:288] T 428edf53f79e4b6d9f4b455cba16e644 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:56.022862 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "428edf53f79e4b6d9f4b455cba16e644" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:56.022953 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "428edf53f79e4b6d9f4b455cba16e644" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:11:56.023110 28021 raft_consensus.cc:3037] T 428edf53f79e4b6d9f4b455cba16e644 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:56.023139 27759 raft_consensus.cc:3037] T 428edf53f79e4b6d9f4b455cba16e644 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:56.025888 27759 raft_consensus.cc:2454] T 428edf53f79e4b6d9f4b455cba16e644 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:56.025892 28021 raft_consensus.cc:2454] T 428edf53f79e4b6d9f4b455cba16e644 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:56.026316 28087 leader_election.cc:302] T 428edf53f79e4b6d9f4b455cba16e644 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:11:56.026644 29971 raft_consensus.cc:2781] T 428edf53f79e4b6d9f4b455cba16e644 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:56.026831 29971 raft_consensus.cc:686] T 428edf53f79e4b6d9f4b455cba16e644 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:11:56.027113 29971 consensus_queue.cc:237] T 428edf53f79e4b6d9f4b455cba16e644 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:56.036252 27619 catalog_manager.cc:5484] T 428edf53f79e4b6d9f4b455cba16e644 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:56.070041 29971 raft_consensus.cc:482] T 188e377fef614ce1801a8f9306d789e1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:56.070243 29971 raft_consensus.cc:504] T 188e377fef614ce1801a8f9306d789e1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:56.070823 29971 leader_election.cc:288] T 188e377fef614ce1801a8f9306d789e1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:56.071449 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188e377fef614ce1801a8f9306d789e1" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:11:56.071424 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188e377fef614ce1801a8f9306d789e1" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:11:56.071677 27759 raft_consensus.cc:2454] T 188e377fef614ce1801a8f9306d789e1 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:56.071697 27890 raft_consensus.cc:2454] T 188e377fef614ce1801a8f9306d789e1 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:56.072178 28087 leader_election.cc:302] T 188e377fef614ce1801a8f9306d789e1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:11:56.072491 29971 raft_consensus.cc:2781] T 188e377fef614ce1801a8f9306d789e1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:56.072577 29971 raft_consensus.cc:482] T 188e377fef614ce1801a8f9306d789e1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:56.072669 29971 raft_consensus.cc:3037] T 188e377fef614ce1801a8f9306d789e1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:56.074724 29971 raft_consensus.cc:504] T 188e377fef614ce1801a8f9306d789e1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:56.075244 29971 leader_election.cc:288] T 188e377fef614ce1801a8f9306d789e1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:56.075726 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188e377fef614ce1801a8f9306d789e1" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:11:56.075718 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188e377fef614ce1801a8f9306d789e1" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:11:56.075910 27759 raft_consensus.cc:3037] T 188e377fef614ce1801a8f9306d789e1 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:56.075912 27890 raft_consensus.cc:3037] T 188e377fef614ce1801a8f9306d789e1 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:56.077786 27890 raft_consensus.cc:2454] T 188e377fef614ce1801a8f9306d789e1 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:56.077822 27759 raft_consensus.cc:2454] T 188e377fef614ce1801a8f9306d789e1 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:56.078212 28088 leader_election.cc:302] T 188e377fef614ce1801a8f9306d789e1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:56.078465 29971 raft_consensus.cc:2781] T 188e377fef614ce1801a8f9306d789e1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:56.078622 29971 raft_consensus.cc:686] T 188e377fef614ce1801a8f9306d789e1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:11:56.078888 29971 consensus_queue.cc:237] T 188e377fef614ce1801a8f9306d789e1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:56.082222 27619 catalog_manager.cc:5484] T 188e377fef614ce1801a8f9306d789e1 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:56.145980 30020 raft_consensus.cc:482] T 271d8e2e56d2468b8f0f9b7835c74a62 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:56.146170 30020 raft_consensus.cc:504] T 271d8e2e56d2468b8f0f9b7835c74a62 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:56.146997 30020 leader_election.cc:288] T 271d8e2e56d2468b8f0f9b7835c74a62 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:56.147428 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "271d8e2e56d2468b8f0f9b7835c74a62" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:56.147598 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "271d8e2e56d2468b8f0f9b7835c74a62" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:11:56.147660 28021 raft_consensus.cc:2454] T 271d8e2e56d2468b8f0f9b7835c74a62 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:11:56.147821 27759 raft_consensus.cc:2454] T 271d8e2e56d2468b8f0f9b7835c74a62 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:11:56.148028 27827 leader_election.cc:302] T 271d8e2e56d2468b8f0f9b7835c74a62 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aad81047009a4ffcbc450b60030583b5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:56.148241 30020 raft_consensus.cc:2781] T 271d8e2e56d2468b8f0f9b7835c74a62 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:56.148350 30020 raft_consensus.cc:482] T 271d8e2e56d2468b8f0f9b7835c74a62 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:56.148440 30020 raft_consensus.cc:3037] T 271d8e2e56d2468b8f0f9b7835c74a62 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:56.150383 30020 raft_consensus.cc:504] T 271d8e2e56d2468b8f0f9b7835c74a62 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:56.150938 30020 leader_election.cc:288] T 271d8e2e56d2468b8f0f9b7835c74a62 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Requested vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:56.151362 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "271d8e2e56d2468b8f0f9b7835c74a62" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:56.151436 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "271d8e2e56d2468b8f0f9b7835c74a62" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:11:56.151582 28022 raft_consensus.cc:3037] T 271d8e2e56d2468b8f0f9b7835c74a62 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:56.151621 27759 raft_consensus.cc:3037] T 271d8e2e56d2468b8f0f9b7835c74a62 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:56.153923 27759 raft_consensus.cc:2454] T 271d8e2e56d2468b8f0f9b7835c74a62 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:11:56.153964 28022 raft_consensus.cc:2454] T 271d8e2e56d2468b8f0f9b7835c74a62 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:11:56.154340 27827 leader_election.cc:302] T 271d8e2e56d2468b8f0f9b7835c74a62 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aad81047009a4ffcbc450b60030583b5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:56.154630 30020 raft_consensus.cc:2781] T 271d8e2e56d2468b8f0f9b7835c74a62 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:56.154776 30020 raft_consensus.cc:686] T 271d8e2e56d2468b8f0f9b7835c74a62 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Becoming Leader. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Running, Role: LEADER
I20240501 08:11:56.155035 30020 consensus_queue.cc:237] T 271d8e2e56d2468b8f0f9b7835c74a62 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:56.158586 27619 catalog_manager.cc:5484] T 271d8e2e56d2468b8f0f9b7835c74a62 P f1525cbc82a24d13b65f136695c903af reported cstate change: term changed from 0 to 1, leader changed from <none> to f1525cbc82a24d13b65f136695c903af (127.21.84.130). New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:56.434583 29971 consensus_queue.cc:1035] T 428edf53f79e4b6d9f4b455cba16e644 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:56.441296 30134 consensus_queue.cc:1035] T 428edf53f79e4b6d9f4b455cba16e644 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:56.573515 30131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:56.573976 30131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:56.582062 30131 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:11:56.590485 30134 consensus_queue.cc:1035] T 188e377fef614ce1801a8f9306d789e1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:56.597625 29872 consensus_queue.cc:1035] T 188e377fef614ce1801a8f9306d789e1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:56.602391 30129 consensus_queue.cc:1035] T 271d8e2e56d2468b8f0f9b7835c74a62 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:56.624747 30129 consensus_queue.cc:1035] T 271d8e2e56d2468b8f0f9b7835c74a62 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_16'
INSERT report
rows total: 2000
time total: 244.518 ms
time per row: 0.122259 ms
I20240501 08:11:57.034389 27617 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalance_test_table_17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:57.045007 27872 tablet_service.cc:1467] Processing CreateTablet for tablet b12559fc416a4f3cbffa018611226bcc (DEFAULT_TABLE table=rebalance_test_table_17 [id=1121d4c7d5444060ac6df36744663e41]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:57.045095 28002 tablet_service.cc:1467] Processing CreateTablet for tablet b12559fc416a4f3cbffa018611226bcc (DEFAULT_TABLE table=rebalance_test_table_17 [id=1121d4c7d5444060ac6df36744663e41]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:57.045496 27872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b12559fc416a4f3cbffa018611226bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:57.045555 28002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b12559fc416a4f3cbffa018611226bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:57.046386 28132 tablet_service.cc:1467] Processing CreateTablet for tablet b12559fc416a4f3cbffa018611226bcc (DEFAULT_TABLE table=rebalance_test_table_17 [id=1121d4c7d5444060ac6df36744663e41]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:57.046659 27740 tablet_service.cc:1467] Processing CreateTablet for tablet 0be6dad1a409426c9a3dc0866505b3ea (DEFAULT_TABLE table=rebalance_test_table_17 [id=1121d4c7d5444060ac6df36744663e41]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:57.046797 28132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b12559fc416a4f3cbffa018611226bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:57.047091 27740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0be6dad1a409426c9a3dc0866505b3ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:57.050804 28000 tablet_service.cc:1467] Processing CreateTablet for tablet 0be6dad1a409426c9a3dc0866505b3ea (DEFAULT_TABLE table=rebalance_test_table_17 [id=1121d4c7d5444060ac6df36744663e41]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:57.051247 28000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0be6dad1a409426c9a3dc0866505b3ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:57.052476 28001 tablet_service.cc:1467] Processing CreateTablet for tablet e40fb0a0d49f44a1a36c33412c630077 (DEFAULT_TABLE table=rebalance_test_table_17 [id=1121d4c7d5444060ac6df36744663e41]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:57.053061 27871 tablet_service.cc:1467] Processing CreateTablet for tablet e40fb0a0d49f44a1a36c33412c630077 (DEFAULT_TABLE table=rebalance_test_table_17 [id=1121d4c7d5444060ac6df36744663e41]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:57.052237 28130 tablet_service.cc:1467] Processing CreateTablet for tablet e40fb0a0d49f44a1a36c33412c630077 (DEFAULT_TABLE table=rebalance_test_table_17 [id=1121d4c7d5444060ac6df36744663e41]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:57.053469 27871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e40fb0a0d49f44a1a36c33412c630077. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:57.054194 28001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e40fb0a0d49f44a1a36c33412c630077. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:57.054287 27870 tablet_service.cc:1467] Processing CreateTablet for tablet 0be6dad1a409426c9a3dc0866505b3ea (DEFAULT_TABLE table=rebalance_test_table_17 [id=1121d4c7d5444060ac6df36744663e41]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:57.054680 27870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0be6dad1a409426c9a3dc0866505b3ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:57.058131 28130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e40fb0a0d49f44a1a36c33412c630077. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:57.066112 30194 tablet_bootstrap.cc:492] T 0be6dad1a409426c9a3dc0866505b3ea P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:57.067631 30195 tablet_bootstrap.cc:492] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:57.069519 30195 tablet_bootstrap.cc:654] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:57.072816 30194 tablet_bootstrap.cc:654] T 0be6dad1a409426c9a3dc0866505b3ea P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:57.072973 30193 tablet_bootstrap.cc:492] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:57.075112 30193 tablet_bootstrap.cc:654] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:57.079522 30198 tablet_bootstrap.cc:492] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:57.081867 30194 tablet_bootstrap.cc:492] T 0be6dad1a409426c9a3dc0866505b3ea P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:57.082124 30194 ts_tablet_manager.cc:1367] T 0be6dad1a409426c9a3dc0866505b3ea P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.004s
I20240501 08:11:57.083004 30195 tablet_bootstrap.cc:492] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:57.083005 30194 raft_consensus.cc:348] T 0be6dad1a409426c9a3dc0866505b3ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:57.083199 30195 ts_tablet_manager.cc:1367] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.004s
I20240501 08:11:57.083329 30194 raft_consensus.cc:374] T 0be6dad1a409426c9a3dc0866505b3ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:57.083478 30194 raft_consensus.cc:729] T 0be6dad1a409426c9a3dc0866505b3ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:57.083575 30198 tablet_bootstrap.cc:654] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:57.083771 30194 consensus_queue.cc:260] T 0be6dad1a409426c9a3dc0866505b3ea P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:57.084550 30194 ts_tablet_manager.cc:1398] T 0be6dad1a409426c9a3dc0866505b3ea P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:11:57.084059 30195 raft_consensus.cc:348] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:57.085824 30195 raft_consensus.cc:374] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:57.086445 30195 raft_consensus.cc:729] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:57.086805 30195 consensus_queue.cc:260] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:57.087610 30195 ts_tablet_manager.cc:1398] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 08:11:57.087996 30195 tablet_bootstrap.cc:492] T e40fb0a0d49f44a1a36c33412c630077 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:57.088858 30193 tablet_bootstrap.cc:492] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:57.089079 30193 ts_tablet_manager.cc:1367] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.000s
I20240501 08:11:57.089915 30195 tablet_bootstrap.cc:654] T e40fb0a0d49f44a1a36c33412c630077 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:57.091797 30193 raft_consensus.cc:348] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:57.092083 30193 raft_consensus.cc:374] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:57.092224 30193 raft_consensus.cc:729] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:57.092516 30193 consensus_queue.cc:260] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:57.095089 30193 ts_tablet_manager.cc:1398] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:11:57.095575 30193 tablet_bootstrap.cc:492] T 0be6dad1a409426c9a3dc0866505b3ea P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:57.097167 30198 tablet_bootstrap.cc:492] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:57.097409 30198 ts_tablet_manager.cc:1367] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.007s
I20240501 08:11:57.099136 30198 raft_consensus.cc:348] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:57.100263 30195 tablet_bootstrap.cc:492] T e40fb0a0d49f44a1a36c33412c630077 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:57.100495 30195 ts_tablet_manager.cc:1367] T e40fb0a0d49f44a1a36c33412c630077 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20240501 08:11:57.101334 30193 tablet_bootstrap.cc:654] T 0be6dad1a409426c9a3dc0866505b3ea P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:57.101449 30195 raft_consensus.cc:348] T e40fb0a0d49f44a1a36c33412c630077 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:57.101792 30195 raft_consensus.cc:374] T e40fb0a0d49f44a1a36c33412c630077 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:57.101907 30195 raft_consensus.cc:729] T e40fb0a0d49f44a1a36c33412c630077 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:57.102294 30195 consensus_queue.cc:260] T e40fb0a0d49f44a1a36c33412c630077 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:57.102336 30198 raft_consensus.cc:374] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:57.102975 30195 ts_tablet_manager.cc:1398] T e40fb0a0d49f44a1a36c33412c630077 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:11:57.103063 30198 raft_consensus.cc:729] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:57.103402 30198 consensus_queue.cc:260] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:57.105396 30198 ts_tablet_manager.cc:1398] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 08:11:57.105969 30198 tablet_bootstrap.cc:492] T b12559fc416a4f3cbffa018611226bcc P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:57.106482 30195 tablet_bootstrap.cc:492] T b12559fc416a4f3cbffa018611226bcc P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:57.108069 30198 tablet_bootstrap.cc:654] T b12559fc416a4f3cbffa018611226bcc P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:57.112713 30195 tablet_bootstrap.cc:654] T b12559fc416a4f3cbffa018611226bcc P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:57.115656 30198 tablet_bootstrap.cc:492] T b12559fc416a4f3cbffa018611226bcc P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:57.115849 30195 tablet_bootstrap.cc:492] T b12559fc416a4f3cbffa018611226bcc P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:57.115999 30195 ts_tablet_manager.cc:1367] T b12559fc416a4f3cbffa018611226bcc P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.001s
I20240501 08:11:57.116001 30198 ts_tablet_manager.cc:1367] T b12559fc416a4f3cbffa018611226bcc P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20240501 08:11:57.116737 30195 raft_consensus.cc:348] T b12559fc416a4f3cbffa018611226bcc P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:57.116966 30195 raft_consensus.cc:374] T b12559fc416a4f3cbffa018611226bcc P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:57.117074 30195 raft_consensus.cc:729] T b12559fc416a4f3cbffa018611226bcc P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:57.117301 30195 consensus_queue.cc:260] T b12559fc416a4f3cbffa018611226bcc P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:57.117085 30198 raft_consensus.cc:348] T b12559fc416a4f3cbffa018611226bcc P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:57.117867 30195 ts_tablet_manager.cc:1398] T b12559fc416a4f3cbffa018611226bcc P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:11:57.118068 30198 raft_consensus.cc:374] T b12559fc416a4f3cbffa018611226bcc P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:57.118211 30198 raft_consensus.cc:729] T b12559fc416a4f3cbffa018611226bcc P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:57.118582 30198 consensus_queue.cc:260] T b12559fc416a4f3cbffa018611226bcc P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:57.120263 30198 ts_tablet_manager.cc:1398] T b12559fc416a4f3cbffa018611226bcc P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 08:11:57.121285 30193 tablet_bootstrap.cc:492] T 0be6dad1a409426c9a3dc0866505b3ea P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:57.121487 30193 ts_tablet_manager.cc:1367] T 0be6dad1a409426c9a3dc0866505b3ea P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.013s
I20240501 08:11:57.122484 30193 raft_consensus.cc:348] T 0be6dad1a409426c9a3dc0866505b3ea P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:57.122730 30193 raft_consensus.cc:374] T 0be6dad1a409426c9a3dc0866505b3ea P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:57.122817 30193 raft_consensus.cc:729] T 0be6dad1a409426c9a3dc0866505b3ea P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:57.123097 30193 consensus_queue.cc:260] T 0be6dad1a409426c9a3dc0866505b3ea P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:57.123602 30193 ts_tablet_manager.cc:1398] T 0be6dad1a409426c9a3dc0866505b3ea P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:57.123950 30193 tablet_bootstrap.cc:492] T e40fb0a0d49f44a1a36c33412c630077 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:57.125806 30193 tablet_bootstrap.cc:654] T e40fb0a0d49f44a1a36c33412c630077 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:57.133172 30193 tablet_bootstrap.cc:492] T e40fb0a0d49f44a1a36c33412c630077 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:57.133345 30193 ts_tablet_manager.cc:1367] T e40fb0a0d49f44a1a36c33412c630077 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20240501 08:11:57.134202 30193 raft_consensus.cc:348] T e40fb0a0d49f44a1a36c33412c630077 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:57.134573 30193 raft_consensus.cc:374] T e40fb0a0d49f44a1a36c33412c630077 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:57.134655 30193 raft_consensus.cc:729] T e40fb0a0d49f44a1a36c33412c630077 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:57.134850 30193 consensus_queue.cc:260] T e40fb0a0d49f44a1a36c33412c630077 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:57.136077 30193 ts_tablet_manager.cc:1398] T e40fb0a0d49f44a1a36c33412c630077 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:57.140286 30179 raft_consensus.cc:482] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:57.140442 30179 raft_consensus.cc:504] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:57.141093 30179 leader_election.cc:288] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:11:57.141499 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6dad1a409426c9a3dc0866505b3ea" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:11:57.141626 28019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6dad1a409426c9a3dc0866505b3ea" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:57.141743 27761 raft_consensus.cc:2454] T 0be6dad1a409426c9a3dc0866505b3ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:11:57.141829 28019 raft_consensus.cc:2454] T 0be6dad1a409426c9a3dc0866505b3ea P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:11:57.142231 27827 leader_election.cc:302] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aad81047009a4ffcbc450b60030583b5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:57.142494 30179 raft_consensus.cc:2781] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:57.142611 30179 raft_consensus.cc:482] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:57.142707 30179 raft_consensus.cc:3037] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:57.144701 30179 raft_consensus.cc:504] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:57.145279 30179 leader_election.cc:288] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Requested vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:11:57.145740 28019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6dad1a409426c9a3dc0866505b3ea" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:57.145756 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be6dad1a409426c9a3dc0866505b3ea" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:11:57.146021 27761 raft_consensus.cc:3037] T 0be6dad1a409426c9a3dc0866505b3ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:57.146041 28019 raft_consensus.cc:3037] T 0be6dad1a409426c9a3dc0866505b3ea P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:57.148779 28019 raft_consensus.cc:2454] T 0be6dad1a409426c9a3dc0866505b3ea P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:11:57.148804 27761 raft_consensus.cc:2454] T 0be6dad1a409426c9a3dc0866505b3ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:11:57.149214 27827 leader_election.cc:302] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:57.149564 30129 raft_consensus.cc:2781] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:57.149714 30129 raft_consensus.cc:686] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Becoming Leader. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Running, Role: LEADER
I20240501 08:11:57.150120 30129 consensus_queue.cc:237] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:11:57.153426 27617 catalog_manager.cc:5484] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af reported cstate change: term changed from 0 to 1, leader changed from <none> to f1525cbc82a24d13b65f136695c903af (127.21.84.130). New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:57.178965 29938 raft_consensus.cc:482] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:57.179140 29938 raft_consensus.cc:504] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:57.180017 29938 leader_election.cc:288] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:11:57.180503 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b12559fc416a4f3cbffa018611226bcc" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:11:57.180511 27891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b12559fc416a4f3cbffa018611226bcc" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:11:57.180721 28151 raft_consensus.cc:2454] T b12559fc416a4f3cbffa018611226bcc P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 0.
I20240501 08:11:57.180747 27891 raft_consensus.cc:2454] T b12559fc416a4f3cbffa018611226bcc P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 0.
I20240501 08:11:57.181212 27955 leader_election.cc:302] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, aad81047009a4ffcbc450b60030583b5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:57.181537 29938 raft_consensus.cc:2781] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:57.181645 29938 raft_consensus.cc:482] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:57.181733 29938 raft_consensus.cc:3037] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:57.183815 29938 raft_consensus.cc:504] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:57.184391 29938 leader_election.cc:288] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Requested vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:11:57.184979 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b12559fc416a4f3cbffa018611226bcc" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:11:57.184990 27891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b12559fc416a4f3cbffa018611226bcc" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:11:57.185204 27891 raft_consensus.cc:3037] T b12559fc416a4f3cbffa018611226bcc P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:57.185216 28151 raft_consensus.cc:3037] T b12559fc416a4f3cbffa018611226bcc P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:57.187240 27891 raft_consensus.cc:2454] T b12559fc416a4f3cbffa018611226bcc P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 1.
I20240501 08:11:57.187433 28151 raft_consensus.cc:2454] T b12559fc416a4f3cbffa018611226bcc P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 1.
I20240501 08:11:57.187747 27958 leader_election.cc:302] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aad81047009a4ffcbc450b60030583b5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:57.188092 29938 raft_consensus.cc:2781] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:57.188467 29938 raft_consensus.cc:686] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Becoming Leader. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Running, Role: LEADER
I20240501 08:11:57.188778 29938 consensus_queue.cc:237] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:57.192301 27617 catalog_manager.cc:5484] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to aad81047009a4ffcbc450b60030583b5 (127.21.84.131). New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:57.291980 29872 raft_consensus.cc:482] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:57.292178 29872 raft_consensus.cc:504] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:57.292835 29872 leader_election.cc:288] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:11:57.293176 27891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40fb0a0d49f44a1a36c33412c630077" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:11:57.293270 28019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40fb0a0d49f44a1a36c33412c630077" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:57.293421 27891 raft_consensus.cc:2454] T e40fb0a0d49f44a1a36c33412c630077 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:57.293501 28019 raft_consensus.cc:2454] T e40fb0a0d49f44a1a36c33412c630077 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:57.293823 28087 leader_election.cc:302] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:57.294099 29872 raft_consensus.cc:2781] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:57.294207 29872 raft_consensus.cc:482] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:57.294306 29872 raft_consensus.cc:3037] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:57.296312 29872 raft_consensus.cc:504] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:57.296937 29872 leader_election.cc:288] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:11:57.297295 27891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40fb0a0d49f44a1a36c33412c630077" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:11:57.297380 28019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40fb0a0d49f44a1a36c33412c630077" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:57.297554 27891 raft_consensus.cc:3037] T e40fb0a0d49f44a1a36c33412c630077 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:57.297567 28019 raft_consensus.cc:3037] T e40fb0a0d49f44a1a36c33412c630077 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:57.300393 28019 raft_consensus.cc:2454] T e40fb0a0d49f44a1a36c33412c630077 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:57.300385 27891 raft_consensus.cc:2454] T e40fb0a0d49f44a1a36c33412c630077 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:57.300870 28087 leader_election.cc:302] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:57.301224 29872 raft_consensus.cc:2781] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:57.301369 29872 raft_consensus.cc:686] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:11:57.301630 29872 consensus_queue.cc:237] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:57.305138 27617 catalog_manager.cc:5484] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:57.685760 29938 consensus_queue.cc:1035] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:57.698695 30210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:57.699172 30210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 08:11:57.700407 30207 consensus_queue.cc:1035] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:57.714054 30210 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:11:57.714660 29971 consensus_queue.cc:1035] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:57.720036 30177 consensus_queue.cc:1035] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:57.759230 30223 consensus_queue.cc:1035] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:57.786671 29971 consensus_queue.cc:1035] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_17'
INSERT report
rows total: 2000
time total: 270.275 ms
time per row: 0.135138 ms
I20240501 08:11:58.209383 27617 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalance_test_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:58.218930 28130 tablet_service.cc:1467] Processing CreateTablet for tablet a87f31b95dda450d8c0cb122a175a9a5 (DEFAULT_TABLE table=rebalance_test_table_18 [id=c750e4f2dd5641909963ec3f64f43082]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:58.219437 28130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a87f31b95dda450d8c0cb122a175a9a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:58.219487 27740 tablet_service.cc:1467] Processing CreateTablet for tablet a87f31b95dda450d8c0cb122a175a9a5 (DEFAULT_TABLE table=rebalance_test_table_18 [id=c750e4f2dd5641909963ec3f64f43082]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:58.219605 27741 tablet_service.cc:1467] Processing CreateTablet for tablet 59a15f7f0dd646908578446fa5532a25 (DEFAULT_TABLE table=rebalance_test_table_18 [id=c750e4f2dd5641909963ec3f64f43082]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:58.219871 27740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a87f31b95dda450d8c0cb122a175a9a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:58.220064 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59a15f7f0dd646908578446fa5532a25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:58.225818 27872 tablet_service.cc:1467] Processing CreateTablet for tablet a87f31b95dda450d8c0cb122a175a9a5 (DEFAULT_TABLE table=rebalance_test_table_18 [id=c750e4f2dd5641909963ec3f64f43082]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:58.225816 28002 tablet_service.cc:1467] Processing CreateTablet for tablet 59a15f7f0dd646908578446fa5532a25 (DEFAULT_TABLE table=rebalance_test_table_18 [id=c750e4f2dd5641909963ec3f64f43082]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:58.226382 27872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a87f31b95dda450d8c0cb122a175a9a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:58.226408 28002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59a15f7f0dd646908578446fa5532a25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:58.236207 27739 tablet_service.cc:1467] Processing CreateTablet for tablet 7fc24ac81fdd48e6baf42e3e675e76d9 (DEFAULT_TABLE table=rebalance_test_table_18 [id=c750e4f2dd5641909963ec3f64f43082]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:58.236682 27739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fc24ac81fdd48e6baf42e3e675e76d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:58.240128 27870 tablet_service.cc:1467] Processing CreateTablet for tablet 7fc24ac81fdd48e6baf42e3e675e76d9 (DEFAULT_TABLE table=rebalance_test_table_18 [id=c750e4f2dd5641909963ec3f64f43082]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:58.240713 27870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fc24ac81fdd48e6baf42e3e675e76d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:58.241791 28132 tablet_service.cc:1467] Processing CreateTablet for tablet 59a15f7f0dd646908578446fa5532a25 (DEFAULT_TABLE table=rebalance_test_table_18 [id=c750e4f2dd5641909963ec3f64f43082]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:58.241942 28131 tablet_service.cc:1467] Processing CreateTablet for tablet 7fc24ac81fdd48e6baf42e3e675e76d9 (DEFAULT_TABLE table=rebalance_test_table_18 [id=c750e4f2dd5641909963ec3f64f43082]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:58.242249 28132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59a15f7f0dd646908578446fa5532a25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:58.246524 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fc24ac81fdd48e6baf42e3e675e76d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:58.251854 30277 tablet_bootstrap.cc:492] T 59a15f7f0dd646908578446fa5532a25 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:58.260932 30277 tablet_bootstrap.cc:654] T 59a15f7f0dd646908578446fa5532a25 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:58.261351 30275 tablet_bootstrap.cc:492] T 59a15f7f0dd646908578446fa5532a25 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:58.263338 30275 tablet_bootstrap.cc:654] T 59a15f7f0dd646908578446fa5532a25 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:58.267678 30276 tablet_bootstrap.cc:492] T a87f31b95dda450d8c0cb122a175a9a5 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:58.272580 30276 tablet_bootstrap.cc:654] T a87f31b95dda450d8c0cb122a175a9a5 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:58.274627 30277 tablet_bootstrap.cc:492] T 59a15f7f0dd646908578446fa5532a25 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:58.274811 30277 ts_tablet_manager.cc:1367] T 59a15f7f0dd646908578446fa5532a25 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.005s
I20240501 08:11:58.275246 30278 tablet_bootstrap.cc:492] T 7fc24ac81fdd48e6baf42e3e675e76d9 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:58.275591 30276 tablet_bootstrap.cc:492] T a87f31b95dda450d8c0cb122a175a9a5 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:58.275837 30276 ts_tablet_manager.cc:1367] T a87f31b95dda450d8c0cb122a175a9a5 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240501 08:11:58.276602 30276 raft_consensus.cc:348] T a87f31b95dda450d8c0cb122a175a9a5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:58.276787 30276 raft_consensus.cc:374] T a87f31b95dda450d8c0cb122a175a9a5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:58.276890 30276 raft_consensus.cc:729] T a87f31b95dda450d8c0cb122a175a9a5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:58.276877 30277 raft_consensus.cc:348] T 59a15f7f0dd646908578446fa5532a25 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:58.277065 30278 tablet_bootstrap.cc:654] T 7fc24ac81fdd48e6baf42e3e675e76d9 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:58.277104 30277 raft_consensus.cc:374] T 59a15f7f0dd646908578446fa5532a25 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:58.277129 30276 consensus_queue.cc:260] T a87f31b95dda450d8c0cb122a175a9a5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:58.277199 30277 raft_consensus.cc:729] T 59a15f7f0dd646908578446fa5532a25 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:58.277796 30276 ts_tablet_manager.cc:1398] T a87f31b95dda450d8c0cb122a175a9a5 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:58.277595 30277 consensus_queue.cc:260] T 59a15f7f0dd646908578446fa5532a25 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:58.278172 30276 tablet_bootstrap.cc:492] T 59a15f7f0dd646908578446fa5532a25 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:58.278789 30277 ts_tablet_manager.cc:1398] T 59a15f7f0dd646908578446fa5532a25 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20240501 08:11:58.280366 30276 tablet_bootstrap.cc:654] T 59a15f7f0dd646908578446fa5532a25 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:58.284713 30278 tablet_bootstrap.cc:492] T 7fc24ac81fdd48e6baf42e3e675e76d9 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:58.285552 30275 tablet_bootstrap.cc:492] T 59a15f7f0dd646908578446fa5532a25 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:58.285758 30275 ts_tablet_manager.cc:1367] T 59a15f7f0dd646908578446fa5532a25 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.000s
I20240501 08:11:58.286655 30275 raft_consensus.cc:348] T 59a15f7f0dd646908578446fa5532a25 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:58.286928 30275 raft_consensus.cc:374] T 59a15f7f0dd646908578446fa5532a25 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:58.287052 30275 raft_consensus.cc:729] T 59a15f7f0dd646908578446fa5532a25 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:58.287276 30275 consensus_queue.cc:260] T 59a15f7f0dd646908578446fa5532a25 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:58.287863 30275 ts_tablet_manager.cc:1398] T 59a15f7f0dd646908578446fa5532a25 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:58.288221 30275 tablet_bootstrap.cc:492] T a87f31b95dda450d8c0cb122a175a9a5 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:58.288547 29650 raft_consensus.cc:482] T 59a15f7f0dd646908578446fa5532a25 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:58.288736 29650 raft_consensus.cc:504] T 59a15f7f0dd646908578446fa5532a25 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:58.288919 30276 tablet_bootstrap.cc:492] T 59a15f7f0dd646908578446fa5532a25 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:58.289078 30276 ts_tablet_manager.cc:1367] T 59a15f7f0dd646908578446fa5532a25 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20240501 08:11:58.289494 29650 leader_election.cc:288] T 59a15f7f0dd646908578446fa5532a25 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:11:58.290040 30275 tablet_bootstrap.cc:654] T a87f31b95dda450d8c0cb122a175a9a5 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:58.289909 30276 raft_consensus.cc:348] T 59a15f7f0dd646908578446fa5532a25 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:58.290169 30276 raft_consensus.cc:374] T 59a15f7f0dd646908578446fa5532a25 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:58.290328 30276 raft_consensus.cc:729] T 59a15f7f0dd646908578446fa5532a25 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:58.290277 28019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a15f7f0dd646908578446fa5532a25" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:58.290359 30278 ts_tablet_manager.cc:1367] T 7fc24ac81fdd48e6baf42e3e675e76d9 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.004s
I20240501 08:11:58.290529 28019 raft_consensus.cc:2454] T 59a15f7f0dd646908578446fa5532a25 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:11:58.290669 30276 consensus_queue.cc:260] T 59a15f7f0dd646908578446fa5532a25 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:58.291191 30278 raft_consensus.cc:348] T 7fc24ac81fdd48e6baf42e3e675e76d9 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:58.291317 30276 ts_tablet_manager.cc:1398] T 59a15f7f0dd646908578446fa5532a25 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:58.291460 30278 raft_consensus.cc:374] T 7fc24ac81fdd48e6baf42e3e675e76d9 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:58.291673 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a15f7f0dd646908578446fa5532a25" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:11:58.291873 28150 raft_consensus.cc:2454] T 59a15f7f0dd646908578446fa5532a25 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:11:58.292253 30276 tablet_bootstrap.cc:492] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:58.292311 27694 leader_election.cc:302] T 59a15f7f0dd646908578446fa5532a25 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:11:58.292500 30278 raft_consensus.cc:729] T 7fc24ac81fdd48e6baf42e3e675e76d9 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:58.292658 30054 raft_consensus.cc:2781] T 59a15f7f0dd646908578446fa5532a25 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:58.292745 30054 raft_consensus.cc:482] T 59a15f7f0dd646908578446fa5532a25 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:58.292855 30054 raft_consensus.cc:3037] T 59a15f7f0dd646908578446fa5532a25 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:58.292747 30278 consensus_queue.cc:260] T 7fc24ac81fdd48e6baf42e3e675e76d9 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:58.293373 30278 ts_tablet_manager.cc:1398] T 7fc24ac81fdd48e6baf42e3e675e76d9 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 08:11:58.293651 30278 tablet_bootstrap.cc:492] T a87f31b95dda450d8c0cb122a175a9a5 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:58.293921 30276 tablet_bootstrap.cc:654] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:58.295959 30054 raft_consensus.cc:504] T 59a15f7f0dd646908578446fa5532a25 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:58.297852 30278 tablet_bootstrap.cc:654] T a87f31b95dda450d8c0cb122a175a9a5 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:58.299734 30275 tablet_bootstrap.cc:492] T a87f31b95dda450d8c0cb122a175a9a5 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:58.299969 30275 ts_tablet_manager.cc:1367] T a87f31b95dda450d8c0cb122a175a9a5 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240501 08:11:58.300841 30275 raft_consensus.cc:348] T a87f31b95dda450d8c0cb122a175a9a5 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:58.301241 30275 raft_consensus.cc:374] T a87f31b95dda450d8c0cb122a175a9a5 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:58.301358 30275 raft_consensus.cc:729] T a87f31b95dda450d8c0cb122a175a9a5 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:58.301638 30275 consensus_queue.cc:260] T a87f31b95dda450d8c0cb122a175a9a5 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:58.302417 30275 ts_tablet_manager.cc:1398] T a87f31b95dda450d8c0cb122a175a9a5 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:11:58.303293 30275 tablet_bootstrap.cc:492] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:58.304826 28019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a15f7f0dd646908578446fa5532a25" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:58.305150 28019 raft_consensus.cc:3037] T 59a15f7f0dd646908578446fa5532a25 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:58.305882 30275 tablet_bootstrap.cc:654] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:58.311213 30278 tablet_bootstrap.cc:492] T a87f31b95dda450d8c0cb122a175a9a5 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:58.311532 30278 ts_tablet_manager.cc:1367] T a87f31b95dda450d8c0cb122a175a9a5 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240501 08:11:58.313105 30278 raft_consensus.cc:348] T a87f31b95dda450d8c0cb122a175a9a5 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:58.313478 30278 raft_consensus.cc:374] T a87f31b95dda450d8c0cb122a175a9a5 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:58.313624 30278 raft_consensus.cc:729] T a87f31b95dda450d8c0cb122a175a9a5 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:58.314078 30278 consensus_queue.cc:260] T a87f31b95dda450d8c0cb122a175a9a5 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:58.315006 30278 ts_tablet_manager.cc:1398] T a87f31b95dda450d8c0cb122a175a9a5 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:58.321193 30275 tablet_bootstrap.cc:492] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:58.321444 30275 ts_tablet_manager.cc:1367] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.000s
I20240501 08:11:58.322358 30275 raft_consensus.cc:348] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:58.322679 30275 raft_consensus.cc:374] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:58.322837 30275 raft_consensus.cc:729] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:58.322932 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a15f7f0dd646908578446fa5532a25" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:11:58.323173 28150 raft_consensus.cc:3037] T 59a15f7f0dd646908578446fa5532a25 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:58.309454 28019 raft_consensus.cc:2454] T 59a15f7f0dd646908578446fa5532a25 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:11:58.324090 27696 leader_election.cc:302] T 59a15f7f0dd646908578446fa5532a25 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:58.324820 29650 raft_consensus.cc:2781] T 59a15f7f0dd646908578446fa5532a25 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:58.324980 29650 raft_consensus.cc:686] T 59a15f7f0dd646908578446fa5532a25 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:11:58.325238 29650 consensus_queue.cc:237] T 59a15f7f0dd646908578446fa5532a25 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:58.326176 28150 raft_consensus.cc:2454] T 59a15f7f0dd646908578446fa5532a25 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:11:58.329732 30054 leader_election.cc:288] T 59a15f7f0dd646908578446fa5532a25 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Requested vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:11:58.331683 30275 consensus_queue.cc:260] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:58.332376 30275 ts_tablet_manager.cc:1398] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 08:11:58.338259 30276 tablet_bootstrap.cc:492] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:58.338438 30276 ts_tablet_manager.cc:1367] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.046s user 0.015s sys 0.023s
I20240501 08:11:58.339269 30276 raft_consensus.cc:348] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:58.339136 27620 catalog_manager.cc:5484] T 59a15f7f0dd646908578446fa5532a25 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:58.339468 30276 raft_consensus.cc:374] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:58.339565 30276 raft_consensus.cc:729] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:58.339788 30276 consensus_queue.cc:260] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:58.340382 30276 ts_tablet_manager.cc:1398] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:11:58.349165 29872 raft_consensus.cc:482] T a87f31b95dda450d8c0cb122a175a9a5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:58.349388 29872 raft_consensus.cc:504] T a87f31b95dda450d8c0cb122a175a9a5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:58.350147 29872 leader_election.cc:288] T a87f31b95dda450d8c0cb122a175a9a5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:58.350881 27891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a87f31b95dda450d8c0cb122a175a9a5" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:11:58.351049 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a87f31b95dda450d8c0cb122a175a9a5" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:11:58.351137 27891 raft_consensus.cc:2454] T a87f31b95dda450d8c0cb122a175a9a5 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:58.351302 27761 raft_consensus.cc:2454] T a87f31b95dda450d8c0cb122a175a9a5 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:58.351575 28088 leader_election.cc:302] T a87f31b95dda450d8c0cb122a175a9a5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:58.351825 29971 raft_consensus.cc:2781] T a87f31b95dda450d8c0cb122a175a9a5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:58.351941 29971 raft_consensus.cc:482] T a87f31b95dda450d8c0cb122a175a9a5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:58.352180 29971 raft_consensus.cc:3037] T a87f31b95dda450d8c0cb122a175a9a5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:58.354367 29971 raft_consensus.cc:504] T a87f31b95dda450d8c0cb122a175a9a5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:58.354956 29971 leader_election.cc:288] T a87f31b95dda450d8c0cb122a175a9a5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:58.355517 27891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a87f31b95dda450d8c0cb122a175a9a5" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:11:58.355505 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a87f31b95dda450d8c0cb122a175a9a5" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:11:58.355722 27891 raft_consensus.cc:3037] T a87f31b95dda450d8c0cb122a175a9a5 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:58.355759 27761 raft_consensus.cc:3037] T a87f31b95dda450d8c0cb122a175a9a5 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:58.358589 27761 raft_consensus.cc:2454] T a87f31b95dda450d8c0cb122a175a9a5 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:58.358601 27891 raft_consensus.cc:2454] T a87f31b95dda450d8c0cb122a175a9a5 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:58.359077 28088 leader_election.cc:302] T a87f31b95dda450d8c0cb122a175a9a5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:58.359414 29971 raft_consensus.cc:2781] T a87f31b95dda450d8c0cb122a175a9a5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:58.359542 29971 raft_consensus.cc:686] T a87f31b95dda450d8c0cb122a175a9a5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:11:58.359843 29971 consensus_queue.cc:237] T a87f31b95dda450d8c0cb122a175a9a5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:58.363207 27617 catalog_manager.cc:5484] T a87f31b95dda450d8c0cb122a175a9a5 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:58.521515 29650 raft_consensus.cc:482] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:58.521696 29650 raft_consensus.cc:504] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:58.522473 29650 leader_election.cc:288] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:11:58.522979 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fc24ac81fdd48e6baf42e3e675e76d9" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:11:58.523231 28150 raft_consensus.cc:2454] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:11:58.523278 27891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fc24ac81fdd48e6baf42e3e675e76d9" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:11:58.523563 27891 raft_consensus.cc:2454] T 7fc24ac81fdd48e6baf42e3e675e76d9 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:11:58.523679 27694 leader_election.cc:302] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:11:58.523963 29650 raft_consensus.cc:2781] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:58.524082 29650 raft_consensus.cc:482] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:58.524169 29650 raft_consensus.cc:3037] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:58.526638 29650 raft_consensus.cc:504] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:58.527211 29650 leader_election.cc:288] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Requested vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:11:58.527737 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fc24ac81fdd48e6baf42e3e675e76d9" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:11:58.527721 27891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fc24ac81fdd48e6baf42e3e675e76d9" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:11:58.527927 28150 raft_consensus.cc:3037] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:58.527942 27891 raft_consensus.cc:3037] T 7fc24ac81fdd48e6baf42e3e675e76d9 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:58.529876 28150 raft_consensus.cc:2454] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:11:58.529889 27891 raft_consensus.cc:2454] T 7fc24ac81fdd48e6baf42e3e675e76d9 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:11:58.530339 27694 leader_election.cc:302] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:11:58.530707 29650 raft_consensus.cc:2781] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:58.530859 29650 raft_consensus.cc:686] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:11:58.531124 29650 consensus_queue.cc:237] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:11:58.534721 27617 catalog_manager.cc:5484] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:58.714730 29650 consensus_queue.cc:1035] T 59a15f7f0dd646908578446fa5532a25 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:58.730275 30289 consensus_queue.cc:1035] T 59a15f7f0dd646908578446fa5532a25 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:58.743306 29971 consensus_queue.cc:1035] T a87f31b95dda450d8c0cb122a175a9a5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:58.748661 29872 consensus_queue.cc:1035] T a87f31b95dda450d8c0cb122a175a9a5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:58.947078 30295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:58.947391 30295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:58.954964 30295 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_18'
I20240501 08:11:59.035141 30291 consensus_queue.cc:1035] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:59.061177 29650 consensus_queue.cc:1035] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:59.099789 27957 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.129:43471, user_credentials={real_user=slave}} blocked reactor thread for 50451.2us
W20240501 08:11:59.152077 27955 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.132:41323, user_credentials={real_user=slave}} blocked reactor thread for 53557.6us
W20240501 08:11:59.295768 28087 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.131:36305, user_credentials={real_user=slave}} blocked reactor thread for 61102.3us
INSERT report
rows total: 2000
time total: 379.232 ms
time per row: 0.189616 ms
I20240501 08:11:59.491712 27617 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalance_test_table_19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:59.501699 28130 tablet_service.cc:1467] Processing CreateTablet for tablet e6fb30f10c754d3a8e73e124ec59768b (DEFAULT_TABLE table=rebalance_test_table_19 [id=4fc7c32c4ee342e5804c1a70b815d952]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:59.501700 27739 tablet_service.cc:1467] Processing CreateTablet for tablet e6fb30f10c754d3a8e73e124ec59768b (DEFAULT_TABLE table=rebalance_test_table_19 [id=4fc7c32c4ee342e5804c1a70b815d952]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:59.502182 27739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6fb30f10c754d3a8e73e124ec59768b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:59.502192 28130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6fb30f10c754d3a8e73e124ec59768b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:59.502339 27741 tablet_service.cc:1467] Processing CreateTablet for tablet 24825c85238b4d618c65d00107135420 (DEFAULT_TABLE table=rebalance_test_table_19 [id=4fc7c32c4ee342e5804c1a70b815d952]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:59.502787 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24825c85238b4d618c65d00107135420. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:59.506783 28000 tablet_service.cc:1467] Processing CreateTablet for tablet 8eb9763a1573455287b7137fe7fecf62 (DEFAULT_TABLE table=rebalance_test_table_19 [id=4fc7c32c4ee342e5804c1a70b815d952]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:59.506832 28001 tablet_service.cc:1467] Processing CreateTablet for tablet e6fb30f10c754d3a8e73e124ec59768b (DEFAULT_TABLE table=rebalance_test_table_19 [id=4fc7c32c4ee342e5804c1a70b815d952]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:59.507211 28000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eb9763a1573455287b7137fe7fecf62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:59.507866 27870 tablet_service.cc:1467] Processing CreateTablet for tablet 24825c85238b4d618c65d00107135420 (DEFAULT_TABLE table=rebalance_test_table_19 [id=4fc7c32c4ee342e5804c1a70b815d952]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:59.508347 27870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24825c85238b4d618c65d00107135420. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:59.510195 28001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6fb30f10c754d3a8e73e124ec59768b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:59.506754 28002 tablet_service.cc:1467] Processing CreateTablet for tablet 24825c85238b4d618c65d00107135420 (DEFAULT_TABLE table=rebalance_test_table_19 [id=4fc7c32c4ee342e5804c1a70b815d952]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:59.514240 28002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24825c85238b4d618c65d00107135420. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:59.518054 27872 tablet_service.cc:1467] Processing CreateTablet for tablet 8eb9763a1573455287b7137fe7fecf62 (DEFAULT_TABLE table=rebalance_test_table_19 [id=4fc7c32c4ee342e5804c1a70b815d952]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:59.518467 27872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eb9763a1573455287b7137fe7fecf62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:59.521512 30365 tablet_bootstrap.cc:492] T 8eb9763a1573455287b7137fe7fecf62 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:59.522493 28131 tablet_service.cc:1467] Processing CreateTablet for tablet 8eb9763a1573455287b7137fe7fecf62 (DEFAULT_TABLE table=rebalance_test_table_19 [id=4fc7c32c4ee342e5804c1a70b815d952]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:59.522910 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eb9763a1573455287b7137fe7fecf62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:59.523321 30365 tablet_bootstrap.cc:654] T 8eb9763a1573455287b7137fe7fecf62 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:59.526111 30364 tablet_bootstrap.cc:492] T e6fb30f10c754d3a8e73e124ec59768b P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:59.528393 30364 tablet_bootstrap.cc:654] T e6fb30f10c754d3a8e73e124ec59768b P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:59.537240 30367 tablet_bootstrap.cc:492] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:59.538076 30364 tablet_bootstrap.cc:492] T e6fb30f10c754d3a8e73e124ec59768b P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:59.538244 30364 ts_tablet_manager.cc:1367] T e6fb30f10c754d3a8e73e124ec59768b P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.002s
I20240501 08:11:59.538885 30365 tablet_bootstrap.cc:492] T 8eb9763a1573455287b7137fe7fecf62 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:59.539101 30365 ts_tablet_manager.cc:1367] T 8eb9763a1573455287b7137fe7fecf62 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.005s
I20240501 08:11:59.539095 30364 raft_consensus.cc:348] T e6fb30f10c754d3a8e73e124ec59768b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:59.539322 30364 raft_consensus.cc:374] T e6fb30f10c754d3a8e73e124ec59768b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:59.539402 30364 raft_consensus.cc:729] T e6fb30f10c754d3a8e73e124ec59768b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:59.539606 30364 consensus_queue.cc:260] T e6fb30f10c754d3a8e73e124ec59768b P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:59.539873 30365 raft_consensus.cc:348] T 8eb9763a1573455287b7137fe7fecf62 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:59.540145 30365 raft_consensus.cc:374] T 8eb9763a1573455287b7137fe7fecf62 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:59.540241 30365 raft_consensus.cc:729] T 8eb9763a1573455287b7137fe7fecf62 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:59.540472 30365 consensus_queue.cc:260] T 8eb9763a1573455287b7137fe7fecf62 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:59.540982 30365 ts_tablet_manager.cc:1398] T 8eb9763a1573455287b7137fe7fecf62 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:11:59.541285 30364 ts_tablet_manager.cc:1398] T e6fb30f10c754d3a8e73e124ec59768b P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:59.543221 30364 tablet_bootstrap.cc:492] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:11:59.543756 30365 tablet_bootstrap.cc:492] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:59.545001 30364 tablet_bootstrap.cc:654] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:59.545600 30365 tablet_bootstrap.cc:654] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:59.554173 30367 tablet_bootstrap.cc:654] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:59.561183 30365 tablet_bootstrap.cc:492] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:59.562321 30365 ts_tablet_manager.cc:1367] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.003s
I20240501 08:11:59.562541 30367 tablet_bootstrap.cc:492] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:59.562739 30367 ts_tablet_manager.cc:1367] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20240501 08:11:59.563567 30367 raft_consensus.cc:348] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:59.563865 30367 raft_consensus.cc:374] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:59.564007 30367 raft_consensus.cc:729] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:59.564327 30367 consensus_queue.cc:260] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:59.565023 30367 ts_tablet_manager.cc:1398] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:11:59.565517 30367 tablet_bootstrap.cc:492] T 8eb9763a1573455287b7137fe7fecf62 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:11:59.567226 30367 tablet_bootstrap.cc:654] T 8eb9763a1573455287b7137fe7fecf62 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:59.567745 30365 raft_consensus.cc:348] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:59.568030 30365 raft_consensus.cc:374] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:59.568125 30365 raft_consensus.cc:729] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:59.568454 30365 consensus_queue.cc:260] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:59.569427 30364 tablet_bootstrap.cc:492] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:11:59.569643 30364 ts_tablet_manager.cc:1367] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.008s
I20240501 08:11:59.570487 30364 raft_consensus.cc:348] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:59.570657 30365 ts_tablet_manager.cc:1398] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240501 08:11:59.570809 30364 raft_consensus.cc:374] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:59.570935 30364 raft_consensus.cc:729] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:11:59.571255 30364 consensus_queue.cc:260] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:59.571895 30364 ts_tablet_manager.cc:1398] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:59.572587 30365 tablet_bootstrap.cc:492] T 24825c85238b4d618c65d00107135420 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:11:59.581614 30367 tablet_bootstrap.cc:492] T 8eb9763a1573455287b7137fe7fecf62 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:11:59.582119 30367 ts_tablet_manager.cc:1367] T 8eb9763a1573455287b7137fe7fecf62 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.004s
I20240501 08:11:59.583102 30365 tablet_bootstrap.cc:654] T 24825c85238b4d618c65d00107135420 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:59.582964 30367 raft_consensus.cc:348] T 8eb9763a1573455287b7137fe7fecf62 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:59.583290 30367 raft_consensus.cc:374] T 8eb9763a1573455287b7137fe7fecf62 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:59.583418 30367 raft_consensus.cc:729] T 8eb9763a1573455287b7137fe7fecf62 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:11:59.583724 30367 consensus_queue.cc:260] T 8eb9763a1573455287b7137fe7fecf62 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:59.584359 30367 ts_tablet_manager.cc:1398] T 8eb9763a1573455287b7137fe7fecf62 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:11:59.593096 30366 tablet_bootstrap.cc:492] T 24825c85238b4d618c65d00107135420 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:59.595788 30366 tablet_bootstrap.cc:654] T 24825c85238b4d618c65d00107135420 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:59.599184 30366 tablet_bootstrap.cc:492] T 24825c85238b4d618c65d00107135420 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:59.599536 30366 ts_tablet_manager.cc:1367] T 24825c85238b4d618c65d00107135420 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 08:11:59.600392 30366 raft_consensus.cc:348] T 24825c85238b4d618c65d00107135420 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:59.600687 30366 raft_consensus.cc:374] T 24825c85238b4d618c65d00107135420 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:59.600785 30366 raft_consensus.cc:729] T 24825c85238b4d618c65d00107135420 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:59.601006 30366 consensus_queue.cc:260] T 24825c85238b4d618c65d00107135420 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:59.603030 30366 ts_tablet_manager.cc:1398] T 24825c85238b4d618c65d00107135420 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:59.603333 30366 tablet_bootstrap.cc:492] T e6fb30f10c754d3a8e73e124ec59768b P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:11:59.603909 30365 tablet_bootstrap.cc:492] T 24825c85238b4d618c65d00107135420 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:11:59.604090 30365 ts_tablet_manager.cc:1367] T 24825c85238b4d618c65d00107135420 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.012s
I20240501 08:11:59.605019 30366 tablet_bootstrap.cc:654] T e6fb30f10c754d3a8e73e124ec59768b P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:59.605129 30365 raft_consensus.cc:348] T 24825c85238b4d618c65d00107135420 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:59.605356 30365 raft_consensus.cc:374] T 24825c85238b4d618c65d00107135420 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:59.605427 30365 raft_consensus.cc:729] T 24825c85238b4d618c65d00107135420 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:11:59.605616 30365 consensus_queue.cc:260] T 24825c85238b4d618c65d00107135420 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:59.606168 30365 ts_tablet_manager.cc:1398] T 24825c85238b4d618c65d00107135420 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 08:11:59.611240 30366 tablet_bootstrap.cc:492] T e6fb30f10c754d3a8e73e124ec59768b P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:11:59.611430 30366 ts_tablet_manager.cc:1367] T e6fb30f10c754d3a8e73e124ec59768b P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240501 08:11:59.612179 30366 raft_consensus.cc:348] T e6fb30f10c754d3a8e73e124ec59768b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:59.612478 30366 raft_consensus.cc:374] T e6fb30f10c754d3a8e73e124ec59768b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:59.612565 30366 raft_consensus.cc:729] T e6fb30f10c754d3a8e73e124ec59768b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:11:59.612823 30366 consensus_queue.cc:260] T e6fb30f10c754d3a8e73e124ec59768b P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:59.613409 30366 ts_tablet_manager.cc:1398] T e6fb30f10c754d3a8e73e124ec59768b P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:59.735602 30293 raft_consensus.cc:482] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:59.735831 30293 raft_consensus.cc:504] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:59.736768 30293 leader_election.cc:288] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:59.737382 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6fb30f10c754d3a8e73e124ec59768b" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:11:59.737453 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6fb30f10c754d3a8e73e124ec59768b" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:11:59.737653 27761 raft_consensus.cc:2454] T e6fb30f10c754d3a8e73e124ec59768b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 0.
I20240501 08:11:59.737658 28150 raft_consensus.cc:2454] T e6fb30f10c754d3a8e73e124ec59768b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 0.
I20240501 08:11:59.738116 27957 leader_election.cc:302] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:59.739020 30293 raft_consensus.cc:2781] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:59.739171 30293 raft_consensus.cc:482] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:59.739262 30293 raft_consensus.cc:3037] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:59.741196 30293 raft_consensus.cc:504] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:59.741775 30293 leader_election.cc:288] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Requested vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:59.742280 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6fb30f10c754d3a8e73e124ec59768b" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:11:59.742341 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6fb30f10c754d3a8e73e124ec59768b" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:11:59.742472 28150 raft_consensus.cc:3037] T e6fb30f10c754d3a8e73e124ec59768b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:59.742528 27761 raft_consensus.cc:3037] T e6fb30f10c754d3a8e73e124ec59768b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:59.744464 27761 raft_consensus.cc:2454] T e6fb30f10c754d3a8e73e124ec59768b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 1.
I20240501 08:11:59.744463 28150 raft_consensus.cc:2454] T e6fb30f10c754d3a8e73e124ec59768b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 1.
I20240501 08:11:59.744904 27955 leader_election.cc:302] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:59.745241 30293 raft_consensus.cc:2781] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:59.745381 30293 raft_consensus.cc:686] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Becoming Leader. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Running, Role: LEADER
I20240501 08:11:59.745633 30293 consensus_queue.cc:237] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:59.749081 27611 catalog_manager.cc:5484] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to aad81047009a4ffcbc450b60030583b5 (127.21.84.131). New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } }
I20240501 08:11:59.790197 30354 raft_consensus.cc:482] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:59.790375 30354 raft_consensus.cc:504] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:59.791007 30354 leader_election.cc:288] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:11:59.791492 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb9763a1573455287b7137fe7fecf62" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:59.791574 27891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb9763a1573455287b7137fe7fecf62" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:11:59.791792 27891 raft_consensus.cc:2454] T 8eb9763a1573455287b7137fe7fecf62 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:59.791793 28022 raft_consensus.cc:2454] T 8eb9763a1573455287b7137fe7fecf62 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:11:59.792268 28088 leader_election.cc:302] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:59.792582 30354 raft_consensus.cc:2781] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:59.792703 30354 raft_consensus.cc:482] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:59.792796 30354 raft_consensus.cc:3037] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:59.795048 30354 raft_consensus.cc:504] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:59.795682 30354 leader_election.cc:288] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:11:59.796172 27891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb9763a1573455287b7137fe7fecf62" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:11:59.796193 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb9763a1573455287b7137fe7fecf62" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:59.796365 27891 raft_consensus.cc:3037] T 8eb9763a1573455287b7137fe7fecf62 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:59.796398 28022 raft_consensus.cc:3037] T 8eb9763a1573455287b7137fe7fecf62 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:59.798568 27891 raft_consensus.cc:2454] T 8eb9763a1573455287b7137fe7fecf62 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:59.798696 28022 raft_consensus.cc:2454] T 8eb9763a1573455287b7137fe7fecf62 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:11:59.799139 28087 leader_election.cc:302] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:11:59.799434 30354 raft_consensus.cc:2781] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:59.799563 30354 raft_consensus.cc:686] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:11:59.799834 30354 consensus_queue.cc:237] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:11:59.803507 27611 catalog_manager.cc:5484] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } }
I20240501 08:11:59.927031 30358 raft_consensus.cc:482] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:59.927235 30358 raft_consensus.cc:504] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:59.927901 30358 leader_election.cc:288] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:59.928392 28019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24825c85238b4d618c65d00107135420" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:11:59.928409 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24825c85238b4d618c65d00107135420" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:11:59.928603 27761 raft_consensus.cc:2454] T 24825c85238b4d618c65d00107135420 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:11:59.928586 28019 raft_consensus.cc:2454] T 24825c85238b4d618c65d00107135420 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:11:59.928942 27827 leader_election.cc:302] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aad81047009a4ffcbc450b60030583b5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:59.929234 30358 raft_consensus.cc:2781] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:59.929349 30358 raft_consensus.cc:482] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:59.929435 30358 raft_consensus.cc:3037] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:59.931603 30358 raft_consensus.cc:504] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:59.932193 30358 leader_election.cc:288] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Requested vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:11:59.932581 28019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24825c85238b4d618c65d00107135420" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:11:59.932768 28019 raft_consensus.cc:3037] T 24825c85238b4d618c65d00107135420 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:59.932736 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24825c85238b4d618c65d00107135420" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:11:59.932925 27761 raft_consensus.cc:3037] T 24825c85238b4d618c65d00107135420 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:59.935114 28019 raft_consensus.cc:2454] T 24825c85238b4d618c65d00107135420 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:11:59.935190 27761 raft_consensus.cc:2454] T 24825c85238b4d618c65d00107135420 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:11:59.935532 27827 leader_election.cc:302] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aad81047009a4ffcbc450b60030583b5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:11:59.935851 30358 raft_consensus.cc:2781] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:59.936003 30358 raft_consensus.cc:686] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Becoming Leader. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Running, Role: LEADER
I20240501 08:11:59.936262 30358 consensus_queue.cc:237] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:11:59.939653 27611 catalog_manager.cc:5484] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af reported cstate change: term changed from 0 to 1, leader changed from <none> to f1525cbc82a24d13b65f136695c903af (127.21.84.130). New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:00.188608 30349 consensus_queue.cc:1035] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:00.204553 30354 consensus_queue.cc:1035] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:00.250363 30293 consensus_queue.cc:1035] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:00.265674 30391 consensus_queue.cc:1035] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:12:00.397151 30382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:00.397468 30382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:00.405200 30382 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_19'
I20240501 08:12:00.465114 30358 consensus_queue.cc:1035] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:00.480196 30420 consensus_queue.cc:1035] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 288.465 ms
time per row: 0.144232 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 08:12:01.837214 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.133:38293
--local_ip_for_outbound_sockets=127.21.84.133
--tserver_master_addrs=127.21.84.190:36935
--webserver_port=38715
--webserver_interface=127.21.84.133
--builtin_ntp_servers=127.21.84.148:39605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 08:12:02.196933 30454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:02.197263 30454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:12:02.197400 30454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:02.206043 30454 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:12:02.206198 30454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:12:02.206348 30454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.133
I20240501 08:12:02.215771 30454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:39605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.133:38293
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.21.84.133
--webserver_port=38715
--tserver_master_addrs=127.21.84.190:36935
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.133
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:12:02.217451 30454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:12:02.219400 30454 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:02.235349 30461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:02.236028 30460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:02.236740 30463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:02.237610 30454 server_base.cc:1034] running on GCE node
I20240501 08:12:02.238440 30454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:12:02.242410 30454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:12:02.244670 30454 hybrid_clock.cc:648] HybridClock initialized: now 1714551122244548 us; error 126 us; skew 500 ppm
I20240501 08:12:02.245172 30454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:02.253763 30454 webserver.cc:460] Webserver started at http://127.21.84.133:38715/ using document root <none> and password file <none>
I20240501 08:12:02.254771 30454 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:02.254921 30454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:02.264473 30454 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.004s sys 0.004s
I20240501 08:12:02.268731 30469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:02.270490 30454 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:02.270666 30454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "31811708c6774037b82df44024b229b6"
format_stamp: "Formatted at 2024-05-01 08:11:27 on dist-test-slave-9cgk"
I20240501 08:12:02.271548 30454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:02.298797 30454 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:12:02.299744 30454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:12:02.300038 30454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:02.301039 30454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:02.302708 30454 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:12:02.302819 30454 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:02.302940 30454 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:12:02.303004 30454 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:02.361969 30454 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.133:38293
I20240501 08:12:02.363288 30583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.133:38293 every 8 connection(s)
I20240501 08:12:02.363613 30454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 08:12:02.370857 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 30454
I20240501 08:12:02.371469 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.134:39933
--local_ip_for_outbound_sockets=127.21.84.134
--tserver_master_addrs=127.21.84.190:36935
--webserver_port=44003
--webserver_interface=127.21.84.134
--builtin_ntp_servers=127.21.84.148:39605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:12:02.382755 30584 heartbeater.cc:347] Connected to a master server at 127.21.84.190:36935
I20240501 08:12:02.383206 30584 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:02.384541 30584 heartbeater.cc:511] Master 127.21.84.190:36935 requested a full tablet report, sending...
I20240501 08:12:02.386873 27616 ts_manager.cc:194] Re-registered known tserver with Master: 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:02.388192 27616 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.133:48803
W20240501 08:12:02.743101 30588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:02.743417 30588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:12:02.743553 30588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:02.752774 30588 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:12:02.752940 30588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:12:02.753094 30588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.134
I20240501 08:12:02.764214 30588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:39605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.134:39933
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.21.84.134
--webserver_port=44003
--tserver_master_addrs=127.21.84.190:36935
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.134
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:12:02.765625 30588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:12:02.767933 30588 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:02.781524 30595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:02.782269 30596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:02.783099 30598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:02.785962 30588 server_base.cc:1034] running on GCE node
I20240501 08:12:02.786646 30588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:12:02.787801 30588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:12:02.789034 30588 hybrid_clock.cc:648] HybridClock initialized: now 1714551122788988 us; error 42 us; skew 500 ppm
I20240501 08:12:02.789355 30588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:02.791751 30588 webserver.cc:460] Webserver started at http://127.21.84.134:44003/ using document root <none> and password file <none>
I20240501 08:12:02.792640 30588 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:02.792820 30588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:02.798370 30588 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20240501 08:12:02.801210 30604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:02.802631 30588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:12:02.802788 30588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "9a022e1a501f4d44a0468054d6c3de47"
format_stamp: "Formatted at 2024-05-01 08:11:27 on dist-test-slave-9cgk"
I20240501 08:12:02.803609 30588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:02.840435 30588 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:12:02.841408 30588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:12:02.841717 30588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:02.842674 30588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:02.844172 30588 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:12:02.844251 30588 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:02.844383 30588 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:12:02.844439 30588 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:02.881870 30588 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.134:39933
I20240501 08:12:02.881901 30717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.134:39933 every 8 connection(s)
I20240501 08:12:02.883484 30588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 08:12:02.886909 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 30588
I20240501 08:12:02.887758 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.135:33503
--local_ip_for_outbound_sockets=127.21.84.135
--tserver_master_addrs=127.21.84.190:36935
--webserver_port=42751
--webserver_interface=127.21.84.135
--builtin_ntp_servers=127.21.84.148:39605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:12:02.898808 30718 heartbeater.cc:347] Connected to a master server at 127.21.84.190:36935
I20240501 08:12:02.899174 30718 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:02.900074 30718 heartbeater.cc:511] Master 127.21.84.190:36935 requested a full tablet report, sending...
I20240501 08:12:02.901619 27616 ts_manager.cc:194] Re-registered known tserver with Master: 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:02.902873 27616 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.134:59341
W20240501 08:12:03.269420 30722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:03.269773 30722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:12:03.269898 30722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:03.278653 30722 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:12:03.278823 30722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:12:03.278955 30722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.135
I20240501 08:12:03.290170 30722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:39605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.135:33503
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.21.84.135
--webserver_port=42751
--tserver_master_addrs=127.21.84.190:36935
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.135
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:12:03.291654 30722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:12:03.294035 30722 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:03.307971 30727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:03.308665 30730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:03.309335 30728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:03.309908 30722 server_base.cc:1034] running on GCE node
I20240501 08:12:03.310551 30722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:12:03.311553 30722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:12:03.312783 30722 hybrid_clock.cc:648] HybridClock initialized: now 1714551123312728 us; error 58 us; skew 500 ppm
I20240501 08:12:03.313123 30722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:03.315744 30722 webserver.cc:460] Webserver started at http://127.21.84.135:42751/ using document root <none> and password file <none>
I20240501 08:12:03.316682 30722 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:03.316844 30722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:03.322729 30722 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20240501 08:12:03.325767 30736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:03.327209 30722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 08:12:03.327373 30722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "64998323a1f4482d8704bf20594d13a5"
format_stamp: "Formatted at 2024-05-01 08:11:28 on dist-test-slave-9cgk"
I20240501 08:12:03.328222 30722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:03.349323 30722 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:12:03.350235 30722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:12:03.350514 30722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:03.351505 30722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:03.353135 30722 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:12:03.353214 30722 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:03.353336 30722 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:12:03.353387 30722 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:03.388964 30722 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.135:33503
I20240501 08:12:03.388983 30849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.135:33503 every 8 connection(s)
I20240501 08:12:03.390727 30722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 08:12:03.392014 30584 heartbeater.cc:503] Master 127.21.84.190:36935 was elected leader, sending a full tablet report...
I20240501 08:12:03.394536 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 30722
I20240501 08:12:03.395216 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.136:46845
--local_ip_for_outbound_sockets=127.21.84.136
--tserver_master_addrs=127.21.84.190:36935
--webserver_port=45935
--webserver_interface=127.21.84.136
--builtin_ntp_servers=127.21.84.148:39605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:12:03.406617 30850 heartbeater.cc:347] Connected to a master server at 127.21.84.190:36935
I20240501 08:12:03.406988 30850 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:03.407902 30850 heartbeater.cc:511] Master 127.21.84.190:36935 requested a full tablet report, sending...
I20240501 08:12:03.409591 27616 ts_manager.cc:194] Re-registered known tserver with Master: 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:03.410516 27616 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.135:53607
W20240501 08:12:03.794163 30853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:03.794493 30853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:12:03.794625 30853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:03.803169 30853 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:12:03.803324 30853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:12:03.803505 30853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.136
I20240501 08:12:03.813725 30853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:39605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.136:46845
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.21.84.136
--webserver_port=45935
--tserver_master_addrs=127.21.84.190:36935
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.136
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:12:03.815390 30853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:12:03.817559 30853 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:03.832345 30861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:03.832645 30860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:03.833303 30863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:03.834321 30853 server_base.cc:1034] running on GCE node
I20240501 08:12:03.834925 30853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:12:03.835899 30853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:12:03.837122 30853 hybrid_clock.cc:648] HybridClock initialized: now 1714551123837047 us; error 56 us; skew 500 ppm
I20240501 08:12:03.837500 30853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:03.840142 30853 webserver.cc:460] Webserver started at http://127.21.84.136:45935/ using document root <none> and password file <none>
I20240501 08:12:03.841095 30853 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:03.841233 30853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:03.846973 30853 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20240501 08:12:03.849867 30869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:03.851456 30853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 08:12:03.851646 30853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "a29cd590999b4f32849ff5689cd53c11"
format_stamp: "Formatted at 2024-05-01 08:11:28 on dist-test-slave-9cgk"
I20240501 08:12:03.853016 30853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:03.867245 30853 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:12:03.868111 30853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:12:03.868415 30853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:03.869354 30853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:03.871659 30853 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:12:03.871763 30853 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:03.871873 30853 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:12:03.871932 30853 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:03.906656 30718 heartbeater.cc:503] Master 127.21.84.190:36935 was elected leader, sending a full tablet report...
I20240501 08:12:03.913468 30853 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.136:46845
I20240501 08:12:03.913499 30982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.136:46845 every 8 connection(s)
I20240501 08:12:03.915067 30853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 08:12:03.921473 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 30853
I20240501 08:12:03.928249 27616 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:03.931926 30983 heartbeater.cc:347] Connected to a master server at 127.21.84.190:36935
I20240501 08:12:03.932291 30983 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:03.933518 30983 heartbeater.cc:511] Master 127.21.84.190:36935 requested a full tablet report, sending...
I20240501 08:12:03.935662 27619 ts_manager.cc:194] Re-registered known tserver with Master: a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:03.936961 27619 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.136:55473
I20240501 08:12:03.977408 30652 tablet_service.cc:1467] Processing CreateTablet for tablet 2c0fa06278b144a281078bc36745a7d8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=449414918ed648569fc292281b36dcd8]), partition=
I20240501 08:12:03.978746 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c0fa06278b144a281078bc36745a7d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:03.981576 30917 tablet_service.cc:1467] Processing CreateTablet for tablet 2c0fa06278b144a281078bc36745a7d8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=449414918ed648569fc292281b36dcd8]), partition=
I20240501 08:12:03.982904 30917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c0fa06278b144a281078bc36745a7d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:03.987707 30517 tablet_service.cc:1467] Processing CreateTablet for tablet 2c0fa06278b144a281078bc36745a7d8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=449414918ed648569fc292281b36dcd8]), partition=
I20240501 08:12:03.988873 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c0fa06278b144a281078bc36745a7d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:04.012696 31002 tablet_bootstrap.cc:492] T 2c0fa06278b144a281078bc36745a7d8 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:04.016147 31004 tablet_bootstrap.cc:492] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:04.017632 31002 tablet_bootstrap.cc:654] T 2c0fa06278b144a281078bc36745a7d8 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:04.018656 31003 tablet_bootstrap.cc:492] T 2c0fa06278b144a281078bc36745a7d8 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:04.020063 31004 tablet_bootstrap.cc:654] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:04.020635 31002 log.cc:824] T 2c0fa06278b144a281078bc36745a7d8 P 9a022e1a501f4d44a0468054d6c3de47: Log is configured to *not* fsync() on all Append() calls
I20240501 08:12:04.021656 31003 tablet_bootstrap.cc:654] T 2c0fa06278b144a281078bc36745a7d8 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:04.021859 31004 log.cc:824] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11: Log is configured to *not* fsync() on all Append() calls
I20240501 08:12:04.022943 31003 log.cc:824] T 2c0fa06278b144a281078bc36745a7d8 P 31811708c6774037b82df44024b229b6: Log is configured to *not* fsync() on all Append() calls
I20240501 08:12:04.024677 31004 tablet_bootstrap.cc:492] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:04.024189 31002 tablet_bootstrap.cc:492] T 2c0fa06278b144a281078bc36745a7d8 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:04.025707 31004 ts_tablet_manager.cc:1367] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240501 08:12:04.026145 31002 ts_tablet_manager.cc:1367] T 2c0fa06278b144a281078bc36745a7d8 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20240501 08:12:04.033404 31003 tablet_bootstrap.cc:492] T 2c0fa06278b144a281078bc36745a7d8 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:04.033725 31003 ts_tablet_manager.cc:1367] T 2c0fa06278b144a281078bc36745a7d8 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20240501 08:12:04.038578 31004 raft_consensus.cc:348] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:04.039093 31004 raft_consensus.cc:374] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:04.039194 31004 raft_consensus.cc:729] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:04.040086 31004 consensus_queue.cc:260] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:04.043639 31003 raft_consensus.cc:348] T 2c0fa06278b144a281078bc36745a7d8 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:04.044055 31003 raft_consensus.cc:374] T 2c0fa06278b144a281078bc36745a7d8 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:04.044137 31003 raft_consensus.cc:729] T 2c0fa06278b144a281078bc36745a7d8 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:04.050979 31003 consensus_queue.cc:260] T 2c0fa06278b144a281078bc36745a7d8 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:04.041301 31002 raft_consensus.cc:348] T 2c0fa06278b144a281078bc36745a7d8 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:04.060079 31002 raft_consensus.cc:374] T 2c0fa06278b144a281078bc36745a7d8 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:04.060284 31002 raft_consensus.cc:729] T 2c0fa06278b144a281078bc36745a7d8 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:04.060364 31004 ts_tablet_manager.cc:1398] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.035s user 0.017s sys 0.001s
I20240501 08:12:04.061254 31002 consensus_queue.cc:260] T 2c0fa06278b144a281078bc36745a7d8 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:04.062644 31003 ts_tablet_manager.cc:1398] T 2c0fa06278b144a281078bc36745a7d8 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.029s user 0.012s sys 0.005s
I20240501 08:12:04.063766 30983 heartbeater.cc:503] Master 127.21.84.190:36935 was elected leader, sending a full tablet report...
I20240501 08:12:04.072518 31002 ts_tablet_manager.cc:1398] T 2c0fa06278b144a281078bc36745a7d8 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.046s user 0.013s sys 0.007s
W20240501 08:12:04.120819 30585 tablet.cc:2369] T 2c0fa06278b144a281078bc36745a7d8 P 31811708c6774037b82df44024b229b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 08:12:04.135844 30719 tablet.cc:2369] T 2c0fa06278b144a281078bc36745a7d8 P 9a022e1a501f4d44a0468054d6c3de47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:12:04.138430 31008 raft_consensus.cc:482] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:04.138679 31008 raft_consensus.cc:504] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:04.140726 31008 leader_election.cc:288] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:04.150089 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c0fa06278b144a281078bc36745a7d8" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:04.150736 30537 raft_consensus.cc:2454] T 2c0fa06278b144a281078bc36745a7d8 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:04.150626 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c0fa06278b144a281078bc36745a7d8" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:04.151161 30672 raft_consensus.cc:2454] T 2c0fa06278b144a281078bc36745a7d8 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:04.151691 30873 leader_election.cc:302] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:04.152312 31008 raft_consensus.cc:2781] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:04.152462 31008 raft_consensus.cc:482] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:04.152546 31008 raft_consensus.cc:3037] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:04.155838 31008 raft_consensus.cc:504] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:04.156548 31008 leader_election.cc:288] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Requested vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:04.157037 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c0fa06278b144a281078bc36745a7d8" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:04.157037 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c0fa06278b144a281078bc36745a7d8" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:04.157258 30537 raft_consensus.cc:3037] T 2c0fa06278b144a281078bc36745a7d8 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:04.157258 30672 raft_consensus.cc:3037] T 2c0fa06278b144a281078bc36745a7d8 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:04.160450 30672 raft_consensus.cc:2454] T 2c0fa06278b144a281078bc36745a7d8 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:04.160436 30537 raft_consensus.cc:2454] T 2c0fa06278b144a281078bc36745a7d8 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:04.160960 30873 leader_election.cc:302] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:04.162833 31008 raft_consensus.cc:2781] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:04.163461 31008 raft_consensus.cc:686] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Becoming Leader. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:12:04.164456 31008 consensus_queue.cc:237] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
W20240501 08:12:04.167080 30984 tablet.cc:2369] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:12:04.173379 27616 catalog_manager.cc:5484] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: term changed from 0 to 1, leader changed from <none> to a29cd590999b4f32849ff5689cd53c11 (127.21.84.136). New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:04.285903 27616 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:04.295536 30537 raft_consensus.cc:1261] T 2c0fa06278b144a281078bc36745a7d8 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:04.295493 30672 raft_consensus.cc:1261] T 2c0fa06278b144a281078bc36745a7d8 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:04.296715 31013 consensus_queue.cc:1035] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:04.306710 27616 catalog_manager.cc:5576] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933) does not have the latest schema for tablet 2c0fa06278b144a281078bc36745a7d8 (table rebalancer_dynamic_table [id=449414918ed648569fc292281b36dcd8]). Expected version 1 got 0
I20240501 08:12:04.306650 31008 consensus_queue.cc:1035] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:04.319217 31022 tablet.cc:1722] T 2c0fa06278b144a281078bc36745a7d8 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:04.321511 27616 catalog_manager.cc:5576] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933) does not have the latest schema for tablet 2c0fa06278b144a281078bc36745a7d8 (table rebalancer_dynamic_table [id=449414918ed648569fc292281b36dcd8]). Expected version 1 got 0
I20240501 08:12:04.322688 31019 tablet.cc:1722] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:04.324339 31019 tablet.cc:1620] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11: MemRowSet was empty: no flush needed.
I20240501 08:12:04.327826 27619 catalog_manager.cc:5576] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293) does not have the latest schema for tablet 2c0fa06278b144a281078bc36745a7d8 (table rebalancer_dynamic_table [id=449414918ed648569fc292281b36dcd8]). Expected version 1 got 0
I20240501 08:12:04.329890 27616 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=449414918ed648569fc292281b36dcd8] alter complete (version 1)
I20240501 08:12:04.331254 31022 tablet.cc:1620] T 2c0fa06278b144a281078bc36745a7d8 P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:04.336846 31020 mvcc.cc:205] Tried to move back new op lower bound from 7022801405103357952 to 7022801404584673280. Current Snapshot: MvccSnapshot[applied={T|T < 7022801405103357952}]
I20240501 08:12:04.343688 31024 tablet.cc:1722] T 2c0fa06278b144a281078bc36745a7d8 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:04.343864 27619 catalog_manager.cc:5576] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293) does not have the latest schema for tablet 2c0fa06278b144a281078bc36745a7d8 (table rebalancer_dynamic_table [id=449414918ed648569fc292281b36dcd8]). Expected version 1 got 0
I20240501 08:12:04.345635 31024 tablet.cc:1620] T 2c0fa06278b144a281078bc36745a7d8 P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:04.413058 30850 heartbeater.cc:503] Master 127.21.84.190:36935 was elected leader, sending a full tablet report...
I20240501 08:12:04.424736 27616 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240501 08:12:04.424664 27619 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_extra_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:04.424947 27616 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240501 08:12:04.431982 27616 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet cd1a890b5d1748b79cdf4bbac02bfecb
I20240501 08:12:04.434587 27619 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:04.435278 27616 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 6cea7a6a22b74bf48285c37ad52c9654
I20240501 08:12:04.435526 28000 tablet_service.cc:1514] Processing DeleteTablet for tablet cd1a890b5d1748b79cdf4bbac02bfecb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:04 UTC) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:04.435719 27616 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet d3febc190f7445c197d8413c0ed82ece
I20240501 08:12:04.436762 27870 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cea7a6a22b74bf48285c37ad52c9654 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:04 UTC) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:04.437341 27871 tablet_service.cc:1514] Processing DeleteTablet for tablet d3febc190f7445c197d8413c0ed82ece with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:04 UTC) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:04.437644 27739 tablet_service.cc:1514] Processing DeleteTablet for tablet d3febc190f7445c197d8413c0ed82ece with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:04 UTC) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:04.437727 28131 tablet_service.cc:1514] Processing DeleteTablet for tablet cd1a890b5d1748b79cdf4bbac02bfecb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:04 UTC) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:04.437916 28130 tablet_service.cc:1514] Processing DeleteTablet for tablet d3febc190f7445c197d8413c0ed82ece with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:04 UTC) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:04.438495 28002 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cea7a6a22b74bf48285c37ad52c9654 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:04 UTC) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:04.437693 27741 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cea7a6a22b74bf48285c37ad52c9654 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:04 UTC) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:04.436762 27872 tablet_service.cc:1514] Processing DeleteTablet for tablet cd1a890b5d1748b79cdf4bbac02bfecb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:04 UTC) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:04.441927 31031 tablet_replica.cc:330] T cd1a890b5d1748b79cdf4bbac02bfecb P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:04.442466 31031 raft_consensus.cc:2229] T cd1a890b5d1748b79cdf4bbac02bfecb P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:04.443061 31031 raft_consensus.cc:2258] T cd1a890b5d1748b79cdf4bbac02bfecb P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:04.444592 31031 ts_tablet_manager.cc:1875] T cd1a890b5d1748b79cdf4bbac02bfecb P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:04.445261 31029 tablet_replica.cc:330] T cd1a890b5d1748b79cdf4bbac02bfecb P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:04.445842 31029 raft_consensus.cc:2229] T cd1a890b5d1748b79cdf4bbac02bfecb P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:04.446539 31029 raft_consensus.cc:2258] T cd1a890b5d1748b79cdf4bbac02bfecb P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:04.449250 31019 tablet.cc:1722] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:04.449820 31019 tablet.cc:1620] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11: MemRowSet was empty: no flush needed.
I20240501 08:12:04.451260 31029 ts_tablet_manager.cc:1875] T cd1a890b5d1748b79cdf4bbac02bfecb P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:04.456828 31032 tablet_replica.cc:330] T d3febc190f7445c197d8413c0ed82ece P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:04.457497 31022 tablet.cc:1722] T 2c0fa06278b144a281078bc36745a7d8 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:04.458250 31022 tablet.cc:1620] T 2c0fa06278b144a281078bc36745a7d8 P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:04.458381 31024 tablet.cc:1722] T 2c0fa06278b144a281078bc36745a7d8 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:04.458875 31024 tablet.cc:1620] T 2c0fa06278b144a281078bc36745a7d8 P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:04.468730 30916 tablet_service.cc:1467] Processing CreateTablet for tablet ecab521101074a7b80b862642059daf2 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=d6a1f8dc8c864cafb8fa47bf4e8e7b3d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:04.469426 30916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecab521101074a7b80b862642059daf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:04.469638 31029 ts_tablet_manager.cc:1888] T cd1a890b5d1748b79cdf4bbac02bfecb P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:04.469798 31029 log.cc:1196] T cd1a890b5d1748b79cdf4bbac02bfecb P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/cd1a890b5d1748b79cdf4bbac02bfecb
I20240501 08:12:04.470374 31029 ts_tablet_manager.cc:1909] T cd1a890b5d1748b79cdf4bbac02bfecb P aad81047009a4ffcbc450b60030583b5: Deleting consensus metadata
I20240501 08:12:04.470539 30917 tablet_service.cc:1467] Processing CreateTablet for tablet 77513d2aea9246b4b37ab54e2a6b496a (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=d6a1f8dc8c864cafb8fa47bf4e8e7b3d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:04.470937 30917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77513d2aea9246b4b37ab54e2a6b496a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:04.473042 31031 ts_tablet_manager.cc:1888] T cd1a890b5d1748b79cdf4bbac02bfecb P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:04.473292 31031 log.cc:1196] T cd1a890b5d1748b79cdf4bbac02bfecb P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/cd1a890b5d1748b79cdf4bbac02bfecb
I20240501 08:12:04.473969 31031 ts_tablet_manager.cc:1909] T cd1a890b5d1748b79cdf4bbac02bfecb P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting consensus metadata
I20240501 08:12:04.477412 30652 tablet_service.cc:1467] Processing CreateTablet for tablet 77513d2aea9246b4b37ab54e2a6b496a (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=d6a1f8dc8c864cafb8fa47bf4e8e7b3d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:04.478121 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77513d2aea9246b4b37ab54e2a6b496a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:04.482134 27616 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=449414918ed648569fc292281b36dcd8] alter complete (version 2)
I20240501 08:12:04.484406 31032 raft_consensus.cc:2229] T d3febc190f7445c197d8413c0ed82ece P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:04.485599 31002 tablet_bootstrap.cc:492] T 77513d2aea9246b4b37ab54e2a6b496a P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:04.487252 31002 tablet_bootstrap.cc:654] T 77513d2aea9246b4b37ab54e2a6b496a P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:04.490047 30517 tablet_service.cc:1467] Processing CreateTablet for tablet ecab521101074a7b80b862642059daf2 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=d6a1f8dc8c864cafb8fa47bf4e8e7b3d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:04.490830 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecab521101074a7b80b862642059daf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:04.494730 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet cd1a890b5d1748b79cdf4bbac02bfecb (table rebalance_test_table_0 [id=5f361e8ecdad4209a6f515928d7198eb]) successfully deleted
I20240501 08:12:04.494988 31031 tablet_replica.cc:330] T d3febc190f7445c197d8413c0ed82ece P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:04.495232 31031 raft_consensus.cc:2229] T d3febc190f7445c197d8413c0ed82ece P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:04.495956 31032 raft_consensus.cc:2258] T d3febc190f7445c197d8413c0ed82ece P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:04.497790 27872 tablet_service.cc:1467] Processing CreateTablet for tablet 19f899c9c362457a9007aa19ec685fd0 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=d6a1f8dc8c864cafb8fa47bf4e8e7b3d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:04.498361 27872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f899c9c362457a9007aa19ec685fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:04.500077 31031 raft_consensus.cc:2258] T d3febc190f7445c197d8413c0ed82ece P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:04.500831 31031 ts_tablet_manager.cc:1875] T d3febc190f7445c197d8413c0ed82ece P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:04.500448 27739 tablet_service.cc:1467] Processing CreateTablet for tablet ecab521101074a7b80b862642059daf2 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=d6a1f8dc8c864cafb8fa47bf4e8e7b3d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:04.501464 27739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecab521101074a7b80b862642059daf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:04.503264 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet cd1a890b5d1748b79cdf4bbac02bfecb (table rebalance_test_table_0 [id=5f361e8ecdad4209a6f515928d7198eb]) successfully deleted
I20240501 08:12:04.504477 31029 tablet_replica.cc:330] T 6cea7a6a22b74bf48285c37ad52c9654 P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:04.504729 31029 raft_consensus.cc:2229] T 6cea7a6a22b74bf48285c37ad52c9654 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:04.504976 31029 raft_consensus.cc:2258] T 6cea7a6a22b74bf48285c37ad52c9654 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:04.505206 28131 tablet_service.cc:1467] Processing CreateTablet for tablet 19f899c9c362457a9007aa19ec685fd0 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=d6a1f8dc8c864cafb8fa47bf4e8e7b3d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:04.505760 31029 ts_tablet_manager.cc:1875] T 6cea7a6a22b74bf48285c37ad52c9654 P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:04.509009 31002 tablet_bootstrap.cc:492] T 77513d2aea9246b4b37ab54e2a6b496a P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:04.509241 31002 ts_tablet_manager.cc:1367] T 77513d2aea9246b4b37ab54e2a6b496a P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.003s
I20240501 08:12:04.511482 31002 raft_consensus.cc:348] T 77513d2aea9246b4b37ab54e2a6b496a P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:04.511723 31002 raft_consensus.cc:374] T 77513d2aea9246b4b37ab54e2a6b496a P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:04.511808 31002 raft_consensus.cc:729] T 77513d2aea9246b4b37ab54e2a6b496a P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:04.512064 31002 consensus_queue.cc:260] T 77513d2aea9246b4b37ab54e2a6b496a P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:04.513365 31002 ts_tablet_manager.cc:1398] T 77513d2aea9246b4b37ab54e2a6b496a P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:04.515425 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f899c9c362457a9007aa19ec685fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:04.523988 31029 ts_tablet_manager.cc:1888] T 6cea7a6a22b74bf48285c37ad52c9654 P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:04.524323 31029 log.cc:1196] T 6cea7a6a22b74bf48285c37ad52c9654 P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/6cea7a6a22b74bf48285c37ad52c9654
I20240501 08:12:04.524370 28000 tablet_service.cc:1467] Processing CreateTablet for tablet 19f899c9c362457a9007aa19ec685fd0 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=d6a1f8dc8c864cafb8fa47bf4e8e7b3d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:04.524842 31029 ts_tablet_manager.cc:1909] T 6cea7a6a22b74bf48285c37ad52c9654 P aad81047009a4ffcbc450b60030583b5: Deleting consensus metadata
I20240501 08:12:04.524987 28000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f899c9c362457a9007aa19ec685fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:04.526850 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet 6cea7a6a22b74bf48285c37ad52c9654 (table rebalance_test_table_0 [id=5f361e8ecdad4209a6f515928d7198eb]) successfully deleted
I20240501 08:12:04.532851 31036 tablet_bootstrap.cc:492] T 19f899c9c362457a9007aa19ec685fd0 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:04.534924 31036 tablet_bootstrap.cc:654] T 19f899c9c362457a9007aa19ec685fd0 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:04.539470 31031 ts_tablet_manager.cc:1888] T d3febc190f7445c197d8413c0ed82ece P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:04.539629 31031 log.cc:1196] T d3febc190f7445c197d8413c0ed82ece P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/d3febc190f7445c197d8413c0ed82ece
I20240501 08:12:04.540089 31031 ts_tablet_manager.cc:1909] T d3febc190f7445c197d8413c0ed82ece P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting consensus metadata
I20240501 08:12:04.540750 31004 tablet_bootstrap.cc:492] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:04.542850 31037 tablet_bootstrap.cc:492] T 19f899c9c362457a9007aa19ec685fd0 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:04.543133 31004 tablet_bootstrap.cc:654] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:04.543532 31003 tablet_bootstrap.cc:492] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:04.545070 31003 tablet_bootstrap.cc:654] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:04.545081 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet d3febc190f7445c197d8413c0ed82ece (table rebalance_test_table_0 [id=5f361e8ecdad4209a6f515928d7198eb]) successfully deleted
I20240501 08:12:04.546545 31032 ts_tablet_manager.cc:1875] T d3febc190f7445c197d8413c0ed82ece P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:04.559581 31030 tablet_replica.cc:330] T 6cea7a6a22b74bf48285c37ad52c9654 P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:04.559671 31037 tablet_bootstrap.cc:654] T 19f899c9c362457a9007aa19ec685fd0 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:04.561547 31004 tablet_bootstrap.cc:492] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:04.560333 31036 tablet_bootstrap.cc:492] T 19f899c9c362457a9007aa19ec685fd0 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:12:04.561870 31004 ts_tablet_manager.cc:1367] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.012s
I20240501 08:12:04.561908 31036 ts_tablet_manager.cc:1367] T 19f899c9c362457a9007aa19ec685fd0 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.000s
I20240501 08:12:04.562417 31039 tablet_bootstrap.cc:492] T 19f899c9c362457a9007aa19ec685fd0 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:04.562861 31030 raft_consensus.cc:2229] T 6cea7a6a22b74bf48285c37ad52c9654 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:04.563127 31004 raft_consensus.cc:348] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:04.563387 31004 raft_consensus.cc:374] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:04.563477 31004 raft_consensus.cc:729] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:04.562976 31036 raft_consensus.cc:348] T 19f899c9c362457a9007aa19ec685fd0 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:04.563728 31036 raft_consensus.cc:374] T 19f899c9c362457a9007aa19ec685fd0 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:04.563848 31036 raft_consensus.cc:729] T 19f899c9c362457a9007aa19ec685fd0 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:12:04.563747 31004 consensus_queue.cc:260] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:04.564075 31036 consensus_queue.cc:260] T 19f899c9c362457a9007aa19ec685fd0 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:04.564759 31036 ts_tablet_manager.cc:1398] T 19f899c9c362457a9007aa19ec685fd0 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:12:04.567467 31030 raft_consensus.cc:2258] T 6cea7a6a22b74bf48285c37ad52c9654 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:04.567714 31004 ts_tablet_manager.cc:1398] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240501 08:12:04.569074 31030 ts_tablet_manager.cc:1875] T 6cea7a6a22b74bf48285c37ad52c9654 P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:04.572069 31037 tablet_bootstrap.cc:492] T 19f899c9c362457a9007aa19ec685fd0 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:12:04.572252 31037 ts_tablet_manager.cc:1367] T 19f899c9c362457a9007aa19ec685fd0 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20240501 08:12:04.572465 31004 tablet_bootstrap.cc:492] T 77513d2aea9246b4b37ab54e2a6b496a P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:04.573076 31037 raft_consensus.cc:348] T 19f899c9c362457a9007aa19ec685fd0 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:04.573335 31037 raft_consensus.cc:374] T 19f899c9c362457a9007aa19ec685fd0 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:04.573426 31037 raft_consensus.cc:729] T 19f899c9c362457a9007aa19ec685fd0 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:12:04.573669 31037 consensus_queue.cc:260] T 19f899c9c362457a9007aa19ec685fd0 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:04.573949 31039 tablet_bootstrap.cc:654] T 19f899c9c362457a9007aa19ec685fd0 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:04.574321 31004 tablet_bootstrap.cc:654] T 77513d2aea9246b4b37ab54e2a6b496a P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:04.576417 31003 tablet_bootstrap.cc:492] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:04.576596 31003 ts_tablet_manager.cc:1367] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.033s user 0.010s sys 0.000s
I20240501 08:12:04.577651 31003 raft_consensus.cc:348] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:04.577879 31003 raft_consensus.cc:374] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:04.577991 31003 raft_consensus.cc:729] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:04.578262 31003 consensus_queue.cc:260] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:04.578888 31003 ts_tablet_manager.cc:1398] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240501 08:12:04.581548 31030 ts_tablet_manager.cc:1888] T 6cea7a6a22b74bf48285c37ad52c9654 P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:04.581745 31030 log.cc:1196] T 6cea7a6a22b74bf48285c37ad52c9654 P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/6cea7a6a22b74bf48285c37ad52c9654
I20240501 08:12:04.586351 31037 ts_tablet_manager.cc:1398] T 19f899c9c362457a9007aa19ec685fd0 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
I20240501 08:12:04.586606 31030 ts_tablet_manager.cc:1909] T 6cea7a6a22b74bf48285c37ad52c9654 P f1525cbc82a24d13b65f136695c903af: Deleting consensus metadata
I20240501 08:12:04.589288 31032 ts_tablet_manager.cc:1888] T d3febc190f7445c197d8413c0ed82ece P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:04.590263 31032 log.cc:1196] T d3febc190f7445c197d8413c0ed82ece P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/d3febc190f7445c197d8413c0ed82ece
I20240501 08:12:04.592115 31032 ts_tablet_manager.cc:1909] T d3febc190f7445c197d8413c0ed82ece P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting consensus metadata
I20240501 08:12:04.593394 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet 6cea7a6a22b74bf48285c37ad52c9654 (table rebalance_test_table_0 [id=5f361e8ecdad4209a6f515928d7198eb]) successfully deleted
I20240501 08:12:04.593974 31032 tablet_replica.cc:330] T 6cea7a6a22b74bf48285c37ad52c9654 P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:04.594159 31039 tablet_bootstrap.cc:492] T 19f899c9c362457a9007aa19ec685fd0 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:12:04.594164 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet d3febc190f7445c197d8413c0ed82ece (table rebalance_test_table_0 [id=5f361e8ecdad4209a6f515928d7198eb]) successfully deleted
I20240501 08:12:04.594368 31039 ts_tablet_manager.cc:1367] T 19f899c9c362457a9007aa19ec685fd0 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.006s
I20240501 08:12:04.595547 27611 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:04.595772 31039 raft_consensus.cc:348] T 19f899c9c362457a9007aa19ec685fd0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:04.595973 31039 raft_consensus.cc:374] T 19f899c9c362457a9007aa19ec685fd0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:04.596051 31039 raft_consensus.cc:729] T 19f899c9c362457a9007aa19ec685fd0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:12:04.596289 31039 consensus_queue.cc:260] T 19f899c9c362457a9007aa19ec685fd0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:04.596949 31039 ts_tablet_manager.cc:1398] T 19f899c9c362457a9007aa19ec685fd0 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:12:04.593061 31030 tablet_replica.cc:330] T d3febc190f7445c197d8413c0ed82ece P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:04.597256 31030 raft_consensus.cc:2229] T d3febc190f7445c197d8413c0ed82ece P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:04.597478 31030 raft_consensus.cc:2258] T d3febc190f7445c197d8413c0ed82ece P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:04.598243 31030 ts_tablet_manager.cc:1875] T d3febc190f7445c197d8413c0ed82ece P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:04.598659 30784 tablet_service.cc:1467] Processing CreateTablet for tablet 77513d2aea9246b4b37ab54e2a6b496a (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=d6a1f8dc8c864cafb8fa47bf4e8e7b3d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:04.600803 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77513d2aea9246b4b37ab54e2a6b496a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:04.602489 31032 raft_consensus.cc:2229] T 6cea7a6a22b74bf48285c37ad52c9654 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:04.602864 31032 raft_consensus.cc:2258] T 6cea7a6a22b74bf48285c37ad52c9654 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:04.603571 31032 ts_tablet_manager.cc:1875] T 6cea7a6a22b74bf48285c37ad52c9654 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:04.606122 31019 tablet.cc:1722] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:04.607144 31004 tablet_bootstrap.cc:492] T 77513d2aea9246b4b37ab54e2a6b496a P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:04.607314 31004 ts_tablet_manager.cc:1367] T 77513d2aea9246b4b37ab54e2a6b496a P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.001s
I20240501 08:12:04.608331 31004 raft_consensus.cc:348] T 77513d2aea9246b4b37ab54e2a6b496a P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:04.608575 31004 raft_consensus.cc:374] T 77513d2aea9246b4b37ab54e2a6b496a P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:04.608649 31004 raft_consensus.cc:729] T 77513d2aea9246b4b37ab54e2a6b496a P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:04.608873 31004 consensus_queue.cc:260] T 77513d2aea9246b4b37ab54e2a6b496a P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:04.609488 31004 ts_tablet_manager.cc:1398] T 77513d2aea9246b4b37ab54e2a6b496a P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:04.613260 31024 tablet.cc:1722] T 2c0fa06278b144a281078bc36745a7d8 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:04.613504 31032 ts_tablet_manager.cc:1888] T 6cea7a6a22b74bf48285c37ad52c9654 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:04.613687 31032 log.cc:1196] T 6cea7a6a22b74bf48285c37ad52c9654 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/6cea7a6a22b74bf48285c37ad52c9654
I20240501 08:12:04.614297 31047 tablet_bootstrap.cc:492] T ecab521101074a7b80b862642059daf2 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:04.618603 31047 tablet_bootstrap.cc:654] T ecab521101074a7b80b862642059daf2 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:04.622812 31022 tablet.cc:1722] T 2c0fa06278b144a281078bc36745a7d8 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:04.622992 31032 ts_tablet_manager.cc:1909] T 6cea7a6a22b74bf48285c37ad52c9654 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting consensus metadata
I20240501 08:12:04.631047 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet 6cea7a6a22b74bf48285c37ad52c9654 (table rebalance_test_table_0 [id=5f361e8ecdad4209a6f515928d7198eb]) successfully deleted
I20240501 08:12:04.631608 27619 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table_new_name [id=449414918ed648569fc292281b36dcd8] alter complete (version 3)
I20240501 08:12:04.642815 31030 ts_tablet_manager.cc:1888] T d3febc190f7445c197d8413c0ed82ece P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:04.643978 31030 log.cc:1196] T d3febc190f7445c197d8413c0ed82ece P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/d3febc190f7445c197d8413c0ed82ece
I20240501 08:12:04.644546 31030 ts_tablet_manager.cc:1909] T d3febc190f7445c197d8413c0ed82ece P f1525cbc82a24d13b65f136695c903af: Deleting consensus metadata
I20240501 08:12:04.645895 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet d3febc190f7445c197d8413c0ed82ece (table rebalance_test_table_0 [id=5f361e8ecdad4209a6f515928d7198eb]) successfully deleted
I20240501 08:12:04.646049 31030 tablet_replica.cc:330] T cd1a890b5d1748b79cdf4bbac02bfecb P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:04.646317 31030 raft_consensus.cc:2229] T cd1a890b5d1748b79cdf4bbac02bfecb P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:04.646823 31030 raft_consensus.cc:2258] T cd1a890b5d1748b79cdf4bbac02bfecb P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:04.648965 31011 raft_consensus.cc:482] T 77513d2aea9246b4b37ab54e2a6b496a P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:04.649200 31011 raft_consensus.cc:504] T 77513d2aea9246b4b37ab54e2a6b496a P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:04.647765 31030 ts_tablet_manager.cc:1875] T cd1a890b5d1748b79cdf4bbac02bfecb P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:04.652038 31011 leader_election.cc:288] T 77513d2aea9246b4b37ab54e2a6b496a P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:04.677558 31030 ts_tablet_manager.cc:1888] T cd1a890b5d1748b79cdf4bbac02bfecb P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:04.679031 31030 log.cc:1196] T cd1a890b5d1748b79cdf4bbac02bfecb P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/cd1a890b5d1748b79cdf4bbac02bfecb
I20240501 08:12:04.680368 31030 ts_tablet_manager.cc:1909] T cd1a890b5d1748b79cdf4bbac02bfecb P f1525cbc82a24d13b65f136695c903af: Deleting consensus metadata
I20240501 08:12:04.683562 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet cd1a890b5d1748b79cdf4bbac02bfecb (table rebalance_test_table_0 [id=5f361e8ecdad4209a6f515928d7198eb]) successfully deleted
W20240501 08:12:04.705809 28198 tablet.cc:2369] T 19f899c9c362457a9007aa19ec685fd0 P 91d11eae5ef6432ca86eb8ed446b6ea6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:12:04.727253 31008 raft_consensus.cc:482] T 77513d2aea9246b4b37ab54e2a6b496a P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:04.727479 31008 raft_consensus.cc:504] T 77513d2aea9246b4b37ab54e2a6b496a P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:04.728507 31008 leader_election.cc:288] T 77513d2aea9246b4b37ab54e2a6b496a P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:04.728839 31009 raft_consensus.cc:482] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:04.729040 31009 raft_consensus.cc:504] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:04.729559 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77513d2aea9246b4b37ab54e2a6b496a" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:04.729825 30672 raft_consensus.cc:2454] T 77513d2aea9246b4b37ab54e2a6b496a P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:04.731017 31009 leader_election.cc:288] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:04.731824 30870 leader_election.cc:302] T 77513d2aea9246b4b37ab54e2a6b496a P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a022e1a501f4d44a0468054d6c3de47, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:04.733845 31008 raft_consensus.cc:2781] T 77513d2aea9246b4b37ab54e2a6b496a P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:04.734014 31008 raft_consensus.cc:482] T 77513d2aea9246b4b37ab54e2a6b496a P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:04.734136 31008 raft_consensus.cc:3037] T 77513d2aea9246b4b37ab54e2a6b496a P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:04.740392 31008 raft_consensus.cc:504] T 77513d2aea9246b4b37ab54e2a6b496a P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
W20240501 08:12:04.743098 28068 tablet.cc:2369] T 19f899c9c362457a9007aa19ec685fd0 P aad81047009a4ffcbc450b60030583b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:12:04.746279 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77513d2aea9246b4b37ab54e2a6b496a" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:04.746603 30672 raft_consensus.cc:3037] T 77513d2aea9246b4b37ab54e2a6b496a P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:04.751627 31008 leader_election.cc:288] T 77513d2aea9246b4b37ab54e2a6b496a P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Requested vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:04.752907 31047 tablet_bootstrap.cc:492] T ecab521101074a7b80b862642059daf2 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:12:04.753062 31052 tablet_bootstrap.cc:492] T 77513d2aea9246b4b37ab54e2a6b496a P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:04.753201 31047 ts_tablet_manager.cc:1367] T ecab521101074a7b80b862642059daf2 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.139s user 0.035s sys 0.030s
I20240501 08:12:04.753973 30672 raft_consensus.cc:2454] T 77513d2aea9246b4b37ab54e2a6b496a P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:04.754220 31047 raft_consensus.cc:348] T ecab521101074a7b80b862642059daf2 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:04.754513 31047 raft_consensus.cc:374] T ecab521101074a7b80b862642059daf2 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:04.754618 30870 leader_election.cc:302] T 77513d2aea9246b4b37ab54e2a6b496a P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a022e1a501f4d44a0468054d6c3de47, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:04.754679 31047 raft_consensus.cc:729] T ecab521101074a7b80b862642059daf2 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:12:04.754952 31047 consensus_queue.cc:260] T ecab521101074a7b80b862642059daf2 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:04.755806 31047 ts_tablet_manager.cc:1398] T ecab521101074a7b80b862642059daf2 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:04.758147 31008 raft_consensus.cc:2781] T 77513d2aea9246b4b37ab54e2a6b496a P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:04.758384 31008 raft_consensus.cc:686] T 77513d2aea9246b4b37ab54e2a6b496a P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Becoming Leader. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:12:04.758759 31008 consensus_queue.cc:237] T 77513d2aea9246b4b37ab54e2a6b496a P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:04.768457 30293 raft_consensus.cc:482] T 19f899c9c362457a9007aa19ec685fd0 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:04.768710 30293 raft_consensus.cc:504] T 19f899c9c362457a9007aa19ec685fd0 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:04.769690 30293 leader_election.cc:288] T 19f899c9c362457a9007aa19ec685fd0 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:12:04.770217 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f899c9c362457a9007aa19ec685fd0" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:12:04.770483 27889 raft_consensus.cc:2454] T 19f899c9c362457a9007aa19ec685fd0 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 0.
I20240501 08:12:04.771005 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f899c9c362457a9007aa19ec685fd0" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:12:04.771291 28151 raft_consensus.cc:2454] T 19f899c9c362457a9007aa19ec685fd0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 0.
I20240501 08:12:04.771447 27958 leader_election.cc:302] T 19f899c9c362457a9007aa19ec685fd0 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aad81047009a4ffcbc450b60030583b5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:04.771814 30293 raft_consensus.cc:2781] T 19f899c9c362457a9007aa19ec685fd0 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:04.771915 30293 raft_consensus.cc:482] T 19f899c9c362457a9007aa19ec685fd0 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:04.771991 30293 raft_consensus.cc:3037] T 19f899c9c362457a9007aa19ec685fd0 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:04.775161 30293 raft_consensus.cc:504] T 19f899c9c362457a9007aa19ec685fd0 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:04.778203 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f899c9c362457a9007aa19ec685fd0" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:12:04.778337 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f899c9c362457a9007aa19ec685fd0" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:12:04.778493 28151 raft_consensus.cc:3037] T 19f899c9c362457a9007aa19ec685fd0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:04.778539 27890 raft_consensus.cc:3037] T 19f899c9c362457a9007aa19ec685fd0 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:04.781574 27890 raft_consensus.cc:2454] T 19f899c9c362457a9007aa19ec685fd0 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 1.
I20240501 08:12:04.785065 27958 leader_election.cc:302] T 19f899c9c362457a9007aa19ec685fd0 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aad81047009a4ffcbc450b60030583b5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:04.787400 30293 leader_election.cc:288] T 19f899c9c362457a9007aa19ec685fd0 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Requested vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:12:04.788465 30293 raft_consensus.cc:2781] T 19f899c9c362457a9007aa19ec685fd0 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:04.789901 30293 raft_consensus.cc:686] T 19f899c9c362457a9007aa19ec685fd0 P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Becoming Leader. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Running, Role: LEADER
I20240501 08:12:04.790918 30293 consensus_queue.cc:237] T 19f899c9c362457a9007aa19ec685fd0 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:04.793220 27619 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:04.793479 27619 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:04.798295 27619 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 2c0fa06278b144a281078bc36745a7d8
I20240501 08:12:04.799945 30916 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c0fa06278b144a281078bc36745a7d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:04 UTC) from {username='slave'} at 127.0.0.1:54036
I20240501 08:12:04.805316 28151 raft_consensus.cc:2454] T 19f899c9c362457a9007aa19ec685fd0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 1.
W20240501 08:12:04.806036 27938 tablet.cc:2369] T 19f899c9c362457a9007aa19ec685fd0 P f1525cbc82a24d13b65f136695c903af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:12:04.807966 31013 raft_consensus.cc:482] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:04.808182 31013 raft_consensus.cc:504] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:04.809177 31013 leader_election.cc:288] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
W20240501 08:12:04.810118 27807 tablet.cc:2369] T ecab521101074a7b80b862642059daf2 P 6331f77dae9d452184e8cd0cbf5f1b43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:12:04.814275 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecab521101074a7b80b862642059daf2" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:04.814592 30537 raft_consensus.cc:2454] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:04.816627 30873 leader_election.cc:302] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:04.817154 31013 raft_consensus.cc:2781] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:04.817265 31013 raft_consensus.cc:482] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:04.817373 31013 raft_consensus.cc:3037] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:04.821640 31013 raft_consensus.cc:504] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:04.825683 27619 catalog_manager.cc:5484] T 19f899c9c362457a9007aa19ec685fd0 P aad81047009a4ffcbc450b60030583b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to aad81047009a4ffcbc450b60030583b5 (127.21.84.131). New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:04.828227 30652 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c0fa06278b144a281078bc36745a7d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:04 UTC) from {username='slave'} at 127.0.0.1:57156
I20240501 08:12:04.831564 30937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77513d2aea9246b4b37ab54e2a6b496a" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:04.838057 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecab521101074a7b80b862642059daf2" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:12:04.838291 27761 raft_consensus.cc:2454] T ecab521101074a7b80b862642059daf2 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:04.838864 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecab521101074a7b80b862642059daf2" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:12:04.838944 30937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecab521101074a7b80b862642059daf2" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:04.839102 27761 raft_consensus.cc:2454] T ecab521101074a7b80b862642059daf2 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 0.
I20240501 08:12:04.840183 30472 leader_election.cc:302] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 6331f77dae9d452184e8cd0cbf5f1b43; no voters:
I20240501 08:12:04.840745 31009 raft_consensus.cc:2781] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:04.840893 31009 raft_consensus.cc:482] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:04.841001 31009 raft_consensus.cc:3037] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:04.842350 30937 raft_consensus.cc:2379] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31811708c6774037b82df44024b229b6 in current term 1: Already voted for candidate a29cd590999b4f32849ff5689cd53c11 in this term.
I20240501 08:12:04.842831 31013 leader_election.cc:288] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Requested vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:04.844053 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecab521101074a7b80b862642059daf2" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:04.844219 31009 raft_consensus.cc:504] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:04.844545 30537 raft_consensus.cc:2379] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a29cd590999b4f32849ff5689cd53c11 in current term 1: Already voted for candidate 31811708c6774037b82df44024b229b6 in this term.
I20240501 08:12:04.830999 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77513d2aea9246b4b37ab54e2a6b496a" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:04.845350 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecab521101074a7b80b862642059daf2" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:12:04.845569 27761 raft_consensus.cc:3037] T ecab521101074a7b80b862642059daf2 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:04.830927 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77513d2aea9246b4b37ab54e2a6b496a" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:04.848695 27761 raft_consensus.cc:2454] T ecab521101074a7b80b862642059daf2 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:04.852185 31009 leader_election.cc:288] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 election: Requested vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:04.854703 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77513d2aea9246b4b37ab54e2a6b496a" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:04.855720 30872 leader_election.cc:302] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, a29cd590999b4f32849ff5689cd53c11; no voters: 31811708c6774037b82df44024b229b6
W20240501 08:12:04.855386 30606 leader_election.cc:341] T 77513d2aea9246b4b37ab54e2a6b496a P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:04.856278 30606 leader_election.cc:302] T 77513d2aea9246b4b37ab54e2a6b496a P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a022e1a501f4d44a0468054d6c3de47; no voters: 64998323a1f4482d8704bf20594d13a5, a29cd590999b4f32849ff5689cd53c11
I20240501 08:12:04.856742 31011 raft_consensus.cc:2726] T 77513d2aea9246b4b37ab54e2a6b496a P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:04.857777 31013 raft_consensus.cc:2781] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:04.858389 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecab521101074a7b80b862642059daf2" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:12:04.859512 27761 raft_consensus.cc:2379] T ecab521101074a7b80b862642059daf2 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 31811708c6774037b82df44024b229b6 in current term 1: Already voted for candidate a29cd590999b4f32849ff5689cd53c11 in this term.
I20240501 08:12:04.861173 31074 tablet_replica.cc:330] T 2c0fa06278b144a281078bc36745a7d8 P 9a022e1a501f4d44a0468054d6c3de47: stopping tablet replica
I20240501 08:12:04.861830 31074 raft_consensus.cc:2229] T 2c0fa06278b144a281078bc36745a7d8 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:04.862288 31074 raft_consensus.cc:2258] T 2c0fa06278b144a281078bc36745a7d8 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:04.863503 31071 tablet_replica.cc:330] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11: stopping tablet replica
I20240501 08:12:04.864118 31071 raft_consensus.cc:2229] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:04.865025 31071 raft_consensus.cc:2258] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 08:12:04.865882 27955 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.132:41323, user_credentials={real_user=slave}} blocked reactor thread for 75293.8us
I20240501 08:12:04.867866 30517 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c0fa06278b144a281078bc36745a7d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:04 UTC) from {username='slave'} at 127.0.0.1:45078
I20240501 08:12:04.867735 27611 catalog_manager.cc:5484] T 77513d2aea9246b4b37ab54e2a6b496a P a29cd590999b4f32849ff5689cd53c11 reported cstate change: term changed from 0 to 1, leader changed from <none> to a29cd590999b4f32849ff5689cd53c11 (127.21.84.136). New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:04.870488 31071 ts_tablet_manager.cc:1875] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:04.871133 31052 tablet_bootstrap.cc:654] T 77513d2aea9246b4b37ab54e2a6b496a P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:04.872467 31052 log.cc:824] T 77513d2aea9246b4b37ab54e2a6b496a P 64998323a1f4482d8704bf20594d13a5: Log is configured to *not* fsync() on all Append() calls
I20240501 08:12:04.878698 31071 ts_tablet_manager.cc:1888] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
W20240501 08:12:04.879745 30871 leader_election.cc:341] T 77513d2aea9246b4b37ab54e2a6b496a P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:04.879844 30937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecab521101074a7b80b862642059daf2" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:04.881640 31076 tablet_replica.cc:330] T 2c0fa06278b144a281078bc36745a7d8 P 31811708c6774037b82df44024b229b6: stopping tablet replica
I20240501 08:12:04.882469 31071 log.cc:1196] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/wals/2c0fa06278b144a281078bc36745a7d8
W20240501 08:12:04.882591 30871 leader_election.cc:341] T 77513d2aea9246b4b37ab54e2a6b496a P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:04.883193 31071 ts_tablet_manager.cc:1909] T 2c0fa06278b144a281078bc36745a7d8 P a29cd590999b4f32849ff5689cd53c11: Deleting consensus metadata
I20240501 08:12:04.883312 31076 raft_consensus.cc:2229] T 2c0fa06278b144a281078bc36745a7d8 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:04.883678 31076 raft_consensus.cc:2258] T 2c0fa06278b144a281078bc36745a7d8 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:04.884160 31074 ts_tablet_manager.cc:1875] T 2c0fa06278b144a281078bc36745a7d8 P 9a022e1a501f4d44a0468054d6c3de47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:04.885062 27606 catalog_manager.cc:4830] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): tablet 2c0fa06278b144a281078bc36745a7d8 (table rebalancer_dynamic_table_new_name [id=449414918ed648569fc292281b36dcd8]) successfully deleted
I20240501 08:12:04.886086 31013 raft_consensus.cc:686] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Becoming Leader. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:12:04.886444 31013 consensus_queue.cc:237] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:04.889513 31076 ts_tablet_manager.cc:1875] T 2c0fa06278b144a281078bc36745a7d8 P 31811708c6774037b82df44024b229b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:04.902962 27611 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:04.917142 27739 tablet_service.cc:1467] Processing CreateTablet for tablet 1abd3e69215d41a78147d9997ba48a3c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=033cf002db9946f69084e909cf5e8a48]), partition=
I20240501 08:12:04.917289 30652 tablet_service.cc:1467] Processing CreateTablet for tablet 1abd3e69215d41a78147d9997ba48a3c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=033cf002db9946f69084e909cf5e8a48]), partition=
I20240501 08:12:04.917567 27739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1abd3e69215d41a78147d9997ba48a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:04.917749 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1abd3e69215d41a78147d9997ba48a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:04.924962 30517 tablet_service.cc:1467] Processing CreateTablet for tablet 1abd3e69215d41a78147d9997ba48a3c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=033cf002db9946f69084e909cf5e8a48]), partition=
I20240501 08:12:04.925631 31047 tablet_bootstrap.cc:492] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:04.927222 31047 tablet_bootstrap.cc:654] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:04.937177 31047 tablet_bootstrap.cc:492] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:12:04.937539 31047 ts_tablet_manager.cc:1367] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.002s
I20240501 08:12:04.938635 31047 raft_consensus.cc:348] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:04.939118 31047 raft_consensus.cc:374] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:04.939260 31047 raft_consensus.cc:729] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:12:04.940073 31047 consensus_queue.cc:260] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:04.940860 31047 ts_tablet_manager.cc:1398] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:04.941605 27619 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240501 08:12:04.941859 27619 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240501 08:12:04.948164 27619 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 0723fe8697e14b6ea9d2ba92213a8a62
I20240501 08:12:04.949337 27619 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet db095644a81a44919887256537e22e70
I20240501 08:12:04.949440 27739 tablet_service.cc:1514] Processing DeleteTablet for tablet 0723fe8697e14b6ea9d2ba92213a8a62 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:04 UTC) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:04.950150 27619 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 0c04c587834c408bb8c43a9b57bf1fe6
I20240501 08:12:04.950358 31032 tablet_replica.cc:330] T 0723fe8697e14b6ea9d2ba92213a8a62 P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:04.950366 28000 tablet_service.cc:1514] Processing DeleteTablet for tablet db095644a81a44919887256537e22e70 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:04 UTC) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:04.950584 31032 raft_consensus.cc:2229] T 0723fe8697e14b6ea9d2ba92213a8a62 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:04.950685 31029 tablet_replica.cc:330] T db095644a81a44919887256537e22e70 P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:04.950968 31032 raft_consensus.cc:2258] T 0723fe8697e14b6ea9d2ba92213a8a62 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:04.950973 31029 raft_consensus.cc:2229] T db095644a81a44919887256537e22e70 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:04.951290 31029 raft_consensus.cc:2258] T db095644a81a44919887256537e22e70 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:04.951773 31032 ts_tablet_manager.cc:1875] T 0723fe8697e14b6ea9d2ba92213a8a62 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:04.952225 31029 ts_tablet_manager.cc:1875] T db095644a81a44919887256537e22e70 P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:04.953246 28131 tablet_service.cc:1514] Processing DeleteTablet for tablet 0723fe8697e14b6ea9d2ba92213a8a62 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:04 UTC) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:04.954175 27739 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c04c587834c408bb8c43a9b57bf1fe6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:04 UTC) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:04.954399 27870 tablet_service.cc:1514] Processing DeleteTablet for tablet 0723fe8697e14b6ea9d2ba92213a8a62 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:04 UTC) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:04.954468 28130 tablet_service.cc:1514] Processing DeleteTablet for tablet db095644a81a44919887256537e22e70 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:04 UTC) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:04.954645 27872 tablet_service.cc:1514] Processing DeleteTablet for tablet db095644a81a44919887256537e22e70 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:04 UTC) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:04.954694 28132 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c04c587834c408bb8c43a9b57bf1fe6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:04 UTC) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:04.954885 31030 tablet_replica.cc:330] T 0723fe8697e14b6ea9d2ba92213a8a62 P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:04.954896 31031 tablet_replica.cc:330] T 0723fe8697e14b6ea9d2ba92213a8a62 P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:04.955081 31031 raft_consensus.cc:2229] T 0723fe8697e14b6ea9d2ba92213a8a62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:04.955093 31030 raft_consensus.cc:2229] T 0723fe8697e14b6ea9d2ba92213a8a62 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:04.955324 31030 raft_consensus.cc:2258] T 0723fe8697e14b6ea9d2ba92213a8a62 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:04.955344 31031 raft_consensus.cc:2258] T 0723fe8697e14b6ea9d2ba92213a8a62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:04.956024 31031 ts_tablet_manager.cc:1875] T 0723fe8697e14b6ea9d2ba92213a8a62 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:04.956041 31030 ts_tablet_manager.cc:1875] T 0723fe8697e14b6ea9d2ba92213a8a62 P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:04.959415 27870 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c04c587834c408bb8c43a9b57bf1fe6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:04 UTC) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:04.963937 31032 ts_tablet_manager.cc:1888] T 0723fe8697e14b6ea9d2ba92213a8a62 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:04.964135 31032 log.cc:1196] T 0723fe8697e14b6ea9d2ba92213a8a62 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/0723fe8697e14b6ea9d2ba92213a8a62
I20240501 08:12:04.964622 31032 ts_tablet_manager.cc:1909] T 0723fe8697e14b6ea9d2ba92213a8a62 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting consensus metadata
I20240501 08:12:04.964843 31030 ts_tablet_manager.cc:1888] T 0723fe8697e14b6ea9d2ba92213a8a62 P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:04.965309 31030 log.cc:1196] T 0723fe8697e14b6ea9d2ba92213a8a62 P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/0723fe8697e14b6ea9d2ba92213a8a62
I20240501 08:12:04.965857 31030 ts_tablet_manager.cc:1909] T 0723fe8697e14b6ea9d2ba92213a8a62 P f1525cbc82a24d13b65f136695c903af: Deleting consensus metadata
I20240501 08:12:04.965811 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet 0723fe8697e14b6ea9d2ba92213a8a62 (table rebalance_test_table_2 [id=9a48ae11d98e46f3ab89d4cbc3fe7020]) successfully deleted
I20240501 08:12:04.966173 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1abd3e69215d41a78147d9997ba48a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:04.966297 31032 tablet_replica.cc:330] T 0c04c587834c408bb8c43a9b57bf1fe6 P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:04.966611 31032 raft_consensus.cc:2229] T 0c04c587834c408bb8c43a9b57bf1fe6 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:04.966905 31032 raft_consensus.cc:2258] T 0c04c587834c408bb8c43a9b57bf1fe6 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:04.967069 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet 0723fe8697e14b6ea9d2ba92213a8a62 (table rebalance_test_table_2 [id=9a48ae11d98e46f3ab89d4cbc3fe7020]) successfully deleted
I20240501 08:12:04.967427 31030 tablet_replica.cc:330] T db095644a81a44919887256537e22e70 P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:04.967682 31032 ts_tablet_manager.cc:1875] T 0c04c587834c408bb8c43a9b57bf1fe6 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:04.967826 31030 raft_consensus.cc:2229] T db095644a81a44919887256537e22e70 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:04.967979 31031 ts_tablet_manager.cc:1888] T 0723fe8697e14b6ea9d2ba92213a8a62 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:04.968099 31031 log.cc:1196] T 0723fe8697e14b6ea9d2ba92213a8a62 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/0723fe8697e14b6ea9d2ba92213a8a62
I20240501 08:12:04.968557 31031 ts_tablet_manager.cc:1909] T 0723fe8697e14b6ea9d2ba92213a8a62 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting consensus metadata
I20240501 08:12:04.968550 31030 raft_consensus.cc:2258] T db095644a81a44919887256537e22e70 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:04.969396 31030 ts_tablet_manager.cc:1875] T db095644a81a44919887256537e22e70 P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:04.969524 31031 tablet_replica.cc:330] T db095644a81a44919887256537e22e70 P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:04.969740 31031 raft_consensus.cc:2229] T db095644a81a44919887256537e22e70 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:04.969987 31031 raft_consensus.cc:2258] T db095644a81a44919887256537e22e70 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:04.972599 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet 0723fe8697e14b6ea9d2ba92213a8a62 (table rebalance_test_table_2 [id=9a48ae11d98e46f3ab89d4cbc3fe7020]) successfully deleted
I20240501 08:12:04.973059 31029 ts_tablet_manager.cc:1888] T db095644a81a44919887256537e22e70 P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:04.973767 31029 log.cc:1196] T db095644a81a44919887256537e22e70 P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/db095644a81a44919887256537e22e70
I20240501 08:12:04.974287 31029 ts_tablet_manager.cc:1909] T db095644a81a44919887256537e22e70 P aad81047009a4ffcbc450b60030583b5: Deleting consensus metadata
I20240501 08:12:04.975039 31031 ts_tablet_manager.cc:1875] T db095644a81a44919887256537e22e70 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:04.977376 31003 tablet_bootstrap.cc:492] T 1abd3e69215d41a78147d9997ba48a3c P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:04.979343 31003 tablet_bootstrap.cc:654] T 1abd3e69215d41a78147d9997ba48a3c P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
W20240501 08:12:04.980055 28087 consensus_peers.cc:485] T 0c04c587834c408bb8c43a9b57bf1fe6 P 91d11eae5ef6432ca86eb8ed446b6ea6 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:04.980742 31002 tablet_bootstrap.cc:492] T 1abd3e69215d41a78147d9997ba48a3c P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:04.982350 31002 tablet_bootstrap.cc:654] T 1abd3e69215d41a78147d9997ba48a3c P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:04.982398 31032 ts_tablet_manager.cc:1888] T 0c04c587834c408bb8c43a9b57bf1fe6 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:04.982544 31032 log.cc:1196] T 0c04c587834c408bb8c43a9b57bf1fe6 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/0c04c587834c408bb8c43a9b57bf1fe6
I20240501 08:12:04.982955 31032 ts_tablet_manager.cc:1909] T 0c04c587834c408bb8c43a9b57bf1fe6 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting consensus metadata
I20240501 08:12:04.984201 31003 tablet_bootstrap.cc:492] T 1abd3e69215d41a78147d9997ba48a3c P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:04.984359 31003 ts_tablet_manager.cc:1367] T 1abd3e69215d41a78147d9997ba48a3c P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20240501 08:12:04.985033 31003 raft_consensus.cc:348] T 1abd3e69215d41a78147d9997ba48a3c P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:04.985255 31003 raft_consensus.cc:374] T 1abd3e69215d41a78147d9997ba48a3c P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:04.985311 31003 raft_consensus.cc:729] T 1abd3e69215d41a78147d9997ba48a3c P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:04.985435 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet 0c04c587834c408bb8c43a9b57bf1fe6 (table rebalance_test_table_2 [id=9a48ae11d98e46f3ab89d4cbc3fe7020]) successfully deleted
I20240501 08:12:04.985605 31003 consensus_queue.cc:260] T 1abd3e69215d41a78147d9997ba48a3c P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:04.987483 31003 ts_tablet_manager.cc:1398] T 1abd3e69215d41a78147d9997ba48a3c P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 08:12:04.990942 31030 ts_tablet_manager.cc:1888] T db095644a81a44919887256537e22e70 P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:04.991094 31030 log.cc:1196] T db095644a81a44919887256537e22e70 P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/db095644a81a44919887256537e22e70
I20240501 08:12:04.991554 31030 ts_tablet_manager.cc:1909] T db095644a81a44919887256537e22e70 P f1525cbc82a24d13b65f136695c903af: Deleting consensus metadata
I20240501 08:12:04.992619 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet db095644a81a44919887256537e22e70 (table rebalance_test_table_2 [id=9a48ae11d98e46f3ab89d4cbc3fe7020]) successfully deleted
I20240501 08:12:04.994086 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet db095644a81a44919887256537e22e70 (table rebalance_test_table_2 [id=9a48ae11d98e46f3ab89d4cbc3fe7020]) successfully deleted
I20240501 08:12:04.999403 31052 tablet_bootstrap.cc:492] T 77513d2aea9246b4b37ab54e2a6b496a P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:04.999972 31052 ts_tablet_manager.cc:1367] T 77513d2aea9246b4b37ab54e2a6b496a P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.247s user 0.045s sys 0.011s
I20240501 08:12:05.004943 31031 ts_tablet_manager.cc:1888] T db095644a81a44919887256537e22e70 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:05.006148 31002 tablet_bootstrap.cc:492] T 1abd3e69215d41a78147d9997ba48a3c P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:05.006417 31002 ts_tablet_manager.cc:1367] T 1abd3e69215d41a78147d9997ba48a3c P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.003s
I20240501 08:12:05.007305 31002 raft_consensus.cc:348] T 1abd3e69215d41a78147d9997ba48a3c P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:05.007591 31002 raft_consensus.cc:374] T 1abd3e69215d41a78147d9997ba48a3c P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:05.007711 31002 raft_consensus.cc:729] T 1abd3e69215d41a78147d9997ba48a3c P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:05.008062 31030 tablet_replica.cc:330] T 0c04c587834c408bb8c43a9b57bf1fe6 P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:05.008087 31002 consensus_queue.cc:260] T 1abd3e69215d41a78147d9997ba48a3c P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:05.008368 31030 raft_consensus.cc:2229] T 0c04c587834c408bb8c43a9b57bf1fe6 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:05.008586 31030 raft_consensus.cc:2258] T 0c04c587834c408bb8c43a9b57bf1fe6 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:05.009377 31030 ts_tablet_manager.cc:1875] T 0c04c587834c408bb8c43a9b57bf1fe6 P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:05.007036 31052 raft_consensus.cc:348] T 77513d2aea9246b4b37ab54e2a6b496a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:05.012029 31052 raft_consensus.cc:374] T 77513d2aea9246b4b37ab54e2a6b496a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:05.012092 31002 ts_tablet_manager.cc:1398] T 1abd3e69215d41a78147d9997ba48a3c P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:12:05.012161 31052 raft_consensus.cc:729] T 77513d2aea9246b4b37ab54e2a6b496a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:05.013106 31052 consensus_queue.cc:260] T 77513d2aea9246b4b37ab54e2a6b496a P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:05.015237 31031 log.cc:1196] T db095644a81a44919887256537e22e70 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/db095644a81a44919887256537e22e70
I20240501 08:12:05.015769 31031 ts_tablet_manager.cc:1909] T db095644a81a44919887256537e22e70 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting consensus metadata
I20240501 08:12:05.016803 31031 tablet_replica.cc:330] T 0c04c587834c408bb8c43a9b57bf1fe6 P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:05.016942 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet db095644a81a44919887256537e22e70 (table rebalance_test_table_2 [id=9a48ae11d98e46f3ab89d4cbc3fe7020]) successfully deleted
I20240501 08:12:05.017113 31031 raft_consensus.cc:2229] T 0c04c587834c408bb8c43a9b57bf1fe6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:05.017500 31031 raft_consensus.cc:2258] T 0c04c587834c408bb8c43a9b57bf1fe6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:05.017827 31074 ts_tablet_manager.cc:1888] T 2c0fa06278b144a281078bc36745a7d8 P 9a022e1a501f4d44a0468054d6c3de47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:05.018425 31031 ts_tablet_manager.cc:1875] T 0c04c587834c408bb8c43a9b57bf1fe6 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:05.019248 31030 ts_tablet_manager.cc:1888] T 0c04c587834c408bb8c43a9b57bf1fe6 P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:05.019421 31030 log.cc:1196] T 0c04c587834c408bb8c43a9b57bf1fe6 P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/0c04c587834c408bb8c43a9b57bf1fe6
I20240501 08:12:05.018107 31074 log.cc:1196] T 2c0fa06278b144a281078bc36745a7d8 P 9a022e1a501f4d44a0468054d6c3de47: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/wals/2c0fa06278b144a281078bc36745a7d8
I20240501 08:12:05.021137 31030 ts_tablet_manager.cc:1909] T 0c04c587834c408bb8c43a9b57bf1fe6 P f1525cbc82a24d13b65f136695c903af: Deleting consensus metadata
I20240501 08:12:05.021641 31074 ts_tablet_manager.cc:1909] T 2c0fa06278b144a281078bc36745a7d8 P 9a022e1a501f4d44a0468054d6c3de47: Deleting consensus metadata
I20240501 08:12:05.022260 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet 0c04c587834c408bb8c43a9b57bf1fe6 (table rebalance_test_table_2 [id=9a48ae11d98e46f3ab89d4cbc3fe7020]) successfully deleted
I20240501 08:12:05.023775 27604 catalog_manager.cc:4830] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): tablet 2c0fa06278b144a281078bc36745a7d8 (table rebalancer_dynamic_table_new_name [id=449414918ed648569fc292281b36dcd8]) successfully deleted
I20240501 08:12:05.025151 31052 ts_tablet_manager.cc:1398] T 77513d2aea9246b4b37ab54e2a6b496a P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.025s user 0.007s sys 0.004s
I20240501 08:12:05.026265 31031 ts_tablet_manager.cc:1888] T 0c04c587834c408bb8c43a9b57bf1fe6 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:05.026398 31031 log.cc:1196] T 0c04c587834c408bb8c43a9b57bf1fe6 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/0c04c587834c408bb8c43a9b57bf1fe6
I20240501 08:12:05.026868 31031 ts_tablet_manager.cc:1909] T 0c04c587834c408bb8c43a9b57bf1fe6 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting consensus metadata
I20240501 08:12:05.028872 31076 ts_tablet_manager.cc:1888] T 2c0fa06278b144a281078bc36745a7d8 P 31811708c6774037b82df44024b229b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:05.029119 31076 log.cc:1196] T 2c0fa06278b144a281078bc36745a7d8 P 31811708c6774037b82df44024b229b6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/wals/2c0fa06278b144a281078bc36745a7d8
I20240501 08:12:05.029773 31076 ts_tablet_manager.cc:1909] T 2c0fa06278b144a281078bc36745a7d8 P 31811708c6774037b82df44024b229b6: Deleting consensus metadata
I20240501 08:12:05.029861 30472 leader_election.cc:302] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31811708c6774037b82df44024b229b6; no voters: 6331f77dae9d452184e8cd0cbf5f1b43, a29cd590999b4f32849ff5689cd53c11
I20240501 08:12:05.030362 31009 raft_consensus.cc:2726] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 08:12:05.031656 27607 catalog_manager.cc:4830] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): tablet 2c0fa06278b144a281078bc36745a7d8 (table rebalancer_dynamic_table_new_name [id=449414918ed648569fc292281b36dcd8]) successfully deleted
I20240501 08:12:05.034621 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet 0c04c587834c408bb8c43a9b57bf1fe6 (table rebalance_test_table_2 [id=9a48ae11d98e46f3ab89d4cbc3fe7020]) successfully deleted
I20240501 08:12:05.035308 27619 catalog_manager.cc:5484] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: term changed from 0 to 1, leader changed from <none> to a29cd590999b4f32849ff5689cd53c11 (127.21.84.136). New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:05.062186 30446 raft_consensus.cc:482] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:05.062377 30446 raft_consensus.cc:504] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:05.063331 30446 leader_election.cc:288] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:05.097466 31009 raft_consensus.cc:482] T 1abd3e69215d41a78147d9997ba48a3c P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:05.097734 31009 raft_consensus.cc:504] T 1abd3e69215d41a78147d9997ba48a3c P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:05.097971 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1abd3e69215d41a78147d9997ba48a3c" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:05.098242 30537 raft_consensus.cc:2454] T 1abd3e69215d41a78147d9997ba48a3c P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:12:05.098393 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1abd3e69215d41a78147d9997ba48a3c" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:05.098614 31009 leader_election.cc:288] T 1abd3e69215d41a78147d9997ba48a3c P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:05.098644 27697 leader_election.cc:302] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 6331f77dae9d452184e8cd0cbf5f1b43; no voters:
I20240501 08:12:05.098670 30672 raft_consensus.cc:2454] T 1abd3e69215d41a78147d9997ba48a3c P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:12:05.099201 30446 raft_consensus.cc:2781] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:05.099323 30446 raft_consensus.cc:482] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:05.099407 30446 raft_consensus.cc:3037] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:05.100301 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1abd3e69215d41a78147d9997ba48a3c" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:12:05.103948 30446 raft_consensus.cc:504] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:05.105625 27761 raft_consensus.cc:2379] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31811708c6774037b82df44024b229b6 in current term 1: Already voted for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in this term.
I20240501 08:12:05.106083 30446 leader_election.cc:288] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Requested vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:05.106487 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1abd3e69215d41a78147d9997ba48a3c" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:05.106676 30537 raft_consensus.cc:3037] T 1abd3e69215d41a78147d9997ba48a3c P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:05.106930 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1abd3e69215d41a78147d9997ba48a3c" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:05.107151 30672 raft_consensus.cc:3037] T 1abd3e69215d41a78147d9997ba48a3c P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:05.109477 30537 raft_consensus.cc:2454] T 1abd3e69215d41a78147d9997ba48a3c P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:12:05.110116 27697 leader_election.cc:302] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 6331f77dae9d452184e8cd0cbf5f1b43; no voters:
I20240501 08:12:05.110527 30446 raft_consensus.cc:2781] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:05.110996 30672 raft_consensus.cc:2454] T 1abd3e69215d41a78147d9997ba48a3c P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:12:05.111182 30446 raft_consensus.cc:686] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:12:05.111629 30446 consensus_queue.cc:237] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:05.114871 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1abd3e69215d41a78147d9997ba48a3c" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:05.115299 30672 raft_consensus.cc:2379] T 1abd3e69215d41a78147d9997ba48a3c P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31811708c6774037b82df44024b229b6 in current term 1: Already voted for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in this term.
I20240501 08:12:05.115969 30470 leader_election.cc:302] T 1abd3e69215d41a78147d9997ba48a3c P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31811708c6774037b82df44024b229b6; no voters: 6331f77dae9d452184e8cd0cbf5f1b43, 9a022e1a501f4d44a0468054d6c3de47
I20240501 08:12:05.116113 27619 catalog_manager.cc:5484] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:05.116477 31009 raft_consensus.cc:2726] T 1abd3e69215d41a78147d9997ba48a3c P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 08:12:05.143472 30851 tablet.cc:2369] T 77513d2aea9246b4b37ab54e2a6b496a P 64998323a1f4482d8704bf20594d13a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:12:05.211891 31082 raft_consensus.cc:482] T 77513d2aea9246b4b37ab54e2a6b496a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:05.212204 31082 raft_consensus.cc:504] T 77513d2aea9246b4b37ab54e2a6b496a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:05.214159 31082 leader_election.cc:288] T 77513d2aea9246b4b37ab54e2a6b496a P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:05.229386 27611 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:05.237906 30804 raft_consensus.cc:3037] T 77513d2aea9246b4b37ab54e2a6b496a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:05.241078 30937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77513d2aea9246b4b37ab54e2a6b496a" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:05.243568 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77513d2aea9246b4b37ab54e2a6b496a" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:05.243880 30672 raft_consensus.cc:2379] T 77513d2aea9246b4b37ab54e2a6b496a P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 64998323a1f4482d8704bf20594d13a5 in current term 1: Already voted for candidate a29cd590999b4f32849ff5689cd53c11 in this term.
I20240501 08:12:05.245179 31013 consensus_queue.cc:1035] T 77513d2aea9246b4b37ab54e2a6b496a P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:05.245743 30737 leader_election.cc:302] T 77513d2aea9246b4b37ab54e2a6b496a P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5; no voters: 9a022e1a501f4d44a0468054d6c3de47, a29cd590999b4f32849ff5689cd53c11
I20240501 08:12:05.249984 31082 raft_consensus.cc:2726] T 77513d2aea9246b4b37ab54e2a6b496a P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:05.253991 31075 consensus_queue.cc:1035] T 77513d2aea9246b4b37ab54e2a6b496a P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:05.259521 30537 raft_consensus.cc:1261] T 1abd3e69215d41a78147d9997ba48a3c P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:05.262707 30671 raft_consensus.cc:1261] T 1abd3e69215d41a78147d9997ba48a3c P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:05.263186 31066 consensus_queue.cc:1035] T 19f899c9c362457a9007aa19ec685fd0 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:05.263679 30446 consensus_queue.cc:1035] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:05.264334 30446 consensus_queue.cc:1035] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:05.273859 31075 consensus_queue.cc:1035] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:05.316840 31066 consensus_queue.cc:1035] T 19f899c9c362457a9007aa19ec685fd0 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:05.341300 31008 consensus_queue.cc:1035] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:05.353521 27739 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 08:12:05.398540 31151 tablet.cc:1722] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:05.400630 27613 catalog_manager.cc:5576] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293) does not have the latest schema for tablet 1abd3e69215d41a78147d9997ba48a3c (table rebalancer_dynamic_table [id=033cf002db9946f69084e909cf5e8a48]). Expected version 1 got 0
I20240501 08:12:05.401422 27611 catalog_manager.cc:5576] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471) does not have the latest schema for tablet 1abd3e69215d41a78147d9997ba48a3c (table rebalancer_dynamic_table [id=033cf002db9946f69084e909cf5e8a48]). Expected version 1 got 0
I20240501 08:12:05.403517 31151 tablet.cc:1620] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43: MemRowSet was empty: no flush needed.
I20240501 08:12:05.404541 31130 mvcc.cc:205] Tried to move back new op lower bound from 7022801409046417408 to 7022801408460046336. Current Snapshot: MvccSnapshot[applied={T|T < 7022801409046417408}]
I20240501 08:12:05.405297 31152 tablet.cc:1722] T 1abd3e69215d41a78147d9997ba48a3c P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:05.406055 31152 tablet.cc:1620] T 1abd3e69215d41a78147d9997ba48a3c P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:05.412640 31153 tablet.cc:1722] T 1abd3e69215d41a78147d9997ba48a3c P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:05.413192 31153 tablet.cc:1620] T 1abd3e69215d41a78147d9997ba48a3c P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:05.417106 27611 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=033cf002db9946f69084e909cf5e8a48] alter complete (version 1)
W20240501 08:12:05.418377 28088 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.130:36503, user_credentials={real_user=slave}} blocked reactor thread for 69584.2us
I20240501 08:12:05.437870 30652 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:05.453881 30517 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:05.456077 27613 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240501 08:12:05.456312 27613 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240501 08:12:05.457691 30916 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:12:05.468942 27613 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 60fb22f7cdeb4c979377537877845c16
I20240501 08:12:05.470655 28132 tablet_service.cc:1514] Processing DeleteTablet for tablet 60fb22f7cdeb4c979377537877845c16 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:05 UTC) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:05.470773 27870 tablet_service.cc:1514] Processing DeleteTablet for tablet 60fb22f7cdeb4c979377537877845c16 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:05 UTC) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:05.470952 31031 tablet_replica.cc:330] T 60fb22f7cdeb4c979377537877845c16 P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:05.471014 31030 tablet_replica.cc:330] T 60fb22f7cdeb4c979377537877845c16 P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:05.471169 31031 raft_consensus.cc:2229] T 60fb22f7cdeb4c979377537877845c16 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:05.471186 31030 raft_consensus.cc:2229] T 60fb22f7cdeb4c979377537877845c16 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:05.471401 31031 raft_consensus.cc:2258] T 60fb22f7cdeb4c979377537877845c16 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:05.471418 31030 raft_consensus.cc:2258] T 60fb22f7cdeb4c979377537877845c16 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:05.472129 31030 ts_tablet_manager.cc:1875] T 60fb22f7cdeb4c979377537877845c16 P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:05.472148 31031 ts_tablet_manager.cc:1875] T 60fb22f7cdeb4c979377537877845c16 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:05.474519 27739 tablet_service.cc:1514] Processing DeleteTablet for tablet 60fb22f7cdeb4c979377537877845c16 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:05 UTC) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:05.474818 31032 tablet_replica.cc:330] T 60fb22f7cdeb4c979377537877845c16 P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:05.475031 31032 raft_consensus.cc:2229] T 60fb22f7cdeb4c979377537877845c16 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:05.475368 31032 raft_consensus.cc:2258] T 60fb22f7cdeb4c979377537877845c16 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:05.476114 31032 ts_tablet_manager.cc:1875] T 60fb22f7cdeb4c979377537877845c16 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:05.477906 28000 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:05.479136 31030 ts_tablet_manager.cc:1888] T 60fb22f7cdeb4c979377537877845c16 P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:05.479347 31030 log.cc:1196] T 60fb22f7cdeb4c979377537877845c16 P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/60fb22f7cdeb4c979377537877845c16
I20240501 08:12:05.479856 31030 ts_tablet_manager.cc:1909] T 60fb22f7cdeb4c979377537877845c16 P f1525cbc82a24d13b65f136695c903af: Deleting consensus metadata
I20240501 08:12:05.481501 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet 60fb22f7cdeb4c979377537877845c16 (table rebalance_test_table_4 [id=9c7af050a91047f384bcfb6909198411]) successfully deleted
I20240501 08:12:05.484747 31031 ts_tablet_manager.cc:1888] T 60fb22f7cdeb4c979377537877845c16 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:05.484921 31031 log.cc:1196] T 60fb22f7cdeb4c979377537877845c16 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/60fb22f7cdeb4c979377537877845c16
I20240501 08:12:05.485459 31031 ts_tablet_manager.cc:1909] T 60fb22f7cdeb4c979377537877845c16 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting consensus metadata
I20240501 08:12:05.486343 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet 60fb22f7cdeb4c979377537877845c16 (table rebalance_test_table_4 [id=9c7af050a91047f384bcfb6909198411]) successfully deleted
I20240501 08:12:05.495203 27613 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 24d7b7f928ad446d8fe0d7cdc507c620
I20240501 08:12:05.496260 28000 tablet_service.cc:1514] Processing DeleteTablet for tablet 24d7b7f928ad446d8fe0d7cdc507c620 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:05 UTC) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:05.496953 27613 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 251e073eb0e44b02bb4ef9cb98aded4b
I20240501 08:12:05.497416 27870 tablet_service.cc:1514] Processing DeleteTablet for tablet 24d7b7f928ad446d8fe0d7cdc507c620 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:05 UTC) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:05.497685 31030 tablet_replica.cc:330] T 24d7b7f928ad446d8fe0d7cdc507c620 P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:05.497680 28002 tablet_service.cc:1514] Processing DeleteTablet for tablet 251e073eb0e44b02bb4ef9cb98aded4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:05 UTC) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:05.497958 31030 raft_consensus.cc:2229] T 24d7b7f928ad446d8fe0d7cdc507c620 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:05.498214 31030 raft_consensus.cc:2258] T 24d7b7f928ad446d8fe0d7cdc507c620 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:05.498245 28132 tablet_service.cc:1514] Processing DeleteTablet for tablet 251e073eb0e44b02bb4ef9cb98aded4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:05 UTC) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:05.499001 31030 ts_tablet_manager.cc:1875] T 24d7b7f928ad446d8fe0d7cdc507c620 P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:05.500481 31031 tablet_replica.cc:330] T 251e073eb0e44b02bb4ef9cb98aded4b P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:05.500725 31031 raft_consensus.cc:2229] T 251e073eb0e44b02bb4ef9cb98aded4b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:05.501138 31031 raft_consensus.cc:2258] T 251e073eb0e44b02bb4ef9cb98aded4b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:05.501942 31031 ts_tablet_manager.cc:1875] T 251e073eb0e44b02bb4ef9cb98aded4b P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:05.502368 27739 tablet_service.cc:1514] Processing DeleteTablet for tablet 24d7b7f928ad446d8fe0d7cdc507c620 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:05 UTC) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:05.502848 27739 tablet_service.cc:1514] Processing DeleteTablet for tablet 251e073eb0e44b02bb4ef9cb98aded4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:05 UTC) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:05.505424 31032 ts_tablet_manager.cc:1888] T 60fb22f7cdeb4c979377537877845c16 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:05.505465 31030 ts_tablet_manager.cc:1888] T 24d7b7f928ad446d8fe0d7cdc507c620 P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:05.505559 31032 log.cc:1196] T 60fb22f7cdeb4c979377537877845c16 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/60fb22f7cdeb4c979377537877845c16
I20240501 08:12:05.505599 31030 log.cc:1196] T 24d7b7f928ad446d8fe0d7cdc507c620 P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/24d7b7f928ad446d8fe0d7cdc507c620
I20240501 08:12:05.506039 31032 ts_tablet_manager.cc:1909] T 60fb22f7cdeb4c979377537877845c16 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting consensus metadata
I20240501 08:12:05.506942 31032 tablet_replica.cc:330] T 24d7b7f928ad446d8fe0d7cdc507c620 P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:05.507699 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet 60fb22f7cdeb4c979377537877845c16 (table rebalance_test_table_4 [id=9c7af050a91047f384bcfb6909198411]) successfully deleted
I20240501 08:12:05.508313 31030 ts_tablet_manager.cc:1909] T 24d7b7f928ad446d8fe0d7cdc507c620 P f1525cbc82a24d13b65f136695c903af: Deleting consensus metadata
I20240501 08:12:05.508661 31032 raft_consensus.cc:2229] T 24d7b7f928ad446d8fe0d7cdc507c620 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:05.509191 31032 raft_consensus.cc:2258] T 24d7b7f928ad446d8fe0d7cdc507c620 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:05.509442 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet 24d7b7f928ad446d8fe0d7cdc507c620 (table rebalance_test_table_4 [id=9c7af050a91047f384bcfb6909198411]) successfully deleted
I20240501 08:12:05.510614 31032 ts_tablet_manager.cc:1875] T 24d7b7f928ad446d8fe0d7cdc507c620 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:05.514473 30784 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:05.529014 27613 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:05.531839 27870 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:12:05.540100 31153 tablet.cc:1722] T 1abd3e69215d41a78147d9997ba48a3c P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:05.540452 31158 tablet_replica.cc:330] T 24d7b7f928ad446d8fe0d7cdc507c620 P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:05.540618 31153 tablet.cc:1620] T 1abd3e69215d41a78147d9997ba48a3c P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:05.541113 31151 tablet.cc:1722] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:05.541579 31151 tablet.cc:1620] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43: MemRowSet was empty: no flush needed.
I20240501 08:12:05.545006 27613 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_extra_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:05.546947 31152 tablet.cc:1722] T 1abd3e69215d41a78147d9997ba48a3c P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:05.547441 31152 tablet.cc:1620] T 1abd3e69215d41a78147d9997ba48a3c P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:05.560619 30784 tablet_service.cc:1467] Processing CreateTablet for tablet bc84d1d413994a76bd6cf42335438755 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=6fb79b6584164a1c9205937b7afef405]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:05.561154 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc84d1d413994a76bd6cf42335438755. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:05.562976 27870 tablet_service.cc:1467] Processing CreateTablet for tablet bc84d1d413994a76bd6cf42335438755 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=6fb79b6584164a1c9205937b7afef405]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:05.563458 27870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc84d1d413994a76bd6cf42335438755. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:05.566468 27611 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=033cf002db9946f69084e909cf5e8a48] alter complete (version 2)
I20240501 08:12:05.569056 30517 tablet_service.cc:1467] Processing CreateTablet for tablet bc84d1d413994a76bd6cf42335438755 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=6fb79b6584164a1c9205937b7afef405]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:05.569525 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc84d1d413994a76bd6cf42335438755. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:05.574541 27872 tablet_service.cc:1467] Processing CreateTablet for tablet 928b70089c234425862c033341c82bad (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=6fb79b6584164a1c9205937b7afef405]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:05.575033 27872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 928b70089c234425862c033341c82bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:05.575666 30783 tablet_service.cc:1467] Processing CreateTablet for tablet 928b70089c234425862c033341c82bad (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=6fb79b6584164a1c9205937b7afef405]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:05.576138 30783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 928b70089c234425862c033341c82bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:05.578137 28132 tablet_service.cc:1467] Processing CreateTablet for tablet 928b70089c234425862c033341c82bad (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=6fb79b6584164a1c9205937b7afef405]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:05.578603 28132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 928b70089c234425862c033341c82bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:05.579959 30652 tablet_service.cc:1467] Processing CreateTablet for tablet 683aa9595f2e456c943fc4d16f22e83c (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=6fb79b6584164a1c9205937b7afef405]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:05.580366 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 683aa9595f2e456c943fc4d16f22e83c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:05.588646 30916 tablet_service.cc:1467] Processing CreateTablet for tablet 683aa9595f2e456c943fc4d16f22e83c (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=6fb79b6584164a1c9205937b7afef405]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:05.589110 30916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 683aa9595f2e456c943fc4d16f22e83c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:05.592532 31160 tablet_bootstrap.cc:492] T 928b70089c234425862c033341c82bad P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:05.594461 31160 tablet_bootstrap.cc:654] T 928b70089c234425862c033341c82bad P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:05.598464 30782 tablet_service.cc:1467] Processing CreateTablet for tablet 683aa9595f2e456c943fc4d16f22e83c (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=6fb79b6584164a1c9205937b7afef405]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:05.599013 30782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 683aa9595f2e456c943fc4d16f22e83c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:05.606258 31161 tablet_bootstrap.cc:492] T 928b70089c234425862c033341c82bad P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:05.615401 31161 tablet_bootstrap.cc:654] T 928b70089c234425862c033341c82bad P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:05.618485 31166 tablet_bootstrap.cc:492] T 683aa9595f2e456c943fc4d16f22e83c P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:05.619232 31164 tablet_bootstrap.cc:492] T 928b70089c234425862c033341c82bad P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:05.619923 31165 tablet_bootstrap.cc:492] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:05.624966 31158 raft_consensus.cc:2229] T 24d7b7f928ad446d8fe0d7cdc507c620 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:05.625416 31158 raft_consensus.cc:2258] T 24d7b7f928ad446d8fe0d7cdc507c620 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:05.625458 31165 tablet_bootstrap.cc:654] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:05.635426 31158 ts_tablet_manager.cc:1875] T 24d7b7f928ad446d8fe0d7cdc507c620 P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:05.638602 31164 tablet_bootstrap.cc:654] T 928b70089c234425862c033341c82bad P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:05.640249 31161 tablet_bootstrap.cc:492] T 928b70089c234425862c033341c82bad P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:05.640539 31161 ts_tablet_manager.cc:1367] T 928b70089c234425862c033341c82bad P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.005s
I20240501 08:12:05.641216 31166 tablet_bootstrap.cc:654] T 683aa9595f2e456c943fc4d16f22e83c P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:05.642215 31161 raft_consensus.cc:348] T 928b70089c234425862c033341c82bad P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:05.642423 31163 tablet_bootstrap.cc:492] T bc84d1d413994a76bd6cf42335438755 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:05.642608 31161 raft_consensus.cc:374] T 928b70089c234425862c033341c82bad P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:05.642758 31161 raft_consensus.cc:729] T 928b70089c234425862c033341c82bad P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:05.643087 31161 consensus_queue.cc:260] T 928b70089c234425862c033341c82bad P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:05.643841 31161 ts_tablet_manager.cc:1398] T 928b70089c234425862c033341c82bad P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:12:05.644173 31161 tablet_bootstrap.cc:492] T 683aa9595f2e456c943fc4d16f22e83c P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:05.645754 31163 tablet_bootstrap.cc:654] T bc84d1d413994a76bd6cf42335438755 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:05.644294 28132 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:12:05.630908 31031 ts_tablet_manager.cc:1888] T 251e073eb0e44b02bb4ef9cb98aded4b P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:05.651383 31031 log.cc:1196] T 251e073eb0e44b02bb4ef9cb98aded4b P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/251e073eb0e44b02bb4ef9cb98aded4b
I20240501 08:12:05.652009 31031 ts_tablet_manager.cc:1909] T 251e073eb0e44b02bb4ef9cb98aded4b P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting consensus metadata
I20240501 08:12:05.652772 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet 251e073eb0e44b02bb4ef9cb98aded4b (table rebalance_test_table_4 [id=9c7af050a91047f384bcfb6909198411]) successfully deleted
I20240501 08:12:05.658398 31032 ts_tablet_manager.cc:1888] T 24d7b7f928ad446d8fe0d7cdc507c620 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:05.658728 31032 log.cc:1196] T 24d7b7f928ad446d8fe0d7cdc507c620 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/24d7b7f928ad446d8fe0d7cdc507c620
I20240501 08:12:05.664007 31166 tablet_bootstrap.cc:492] T 683aa9595f2e456c943fc4d16f22e83c P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:05.664324 31166 ts_tablet_manager.cc:1367] T 683aa9595f2e456c943fc4d16f22e83c P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.001s
I20240501 08:12:05.664829 31082 raft_consensus.cc:482] T 928b70089c234425862c033341c82bad P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:05.665035 31082 raft_consensus.cc:504] T 928b70089c234425862c033341c82bad P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:05.665917 31082 leader_election.cc:288] T 928b70089c234425862c033341c82bad P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:05.659507 31032 ts_tablet_manager.cc:1909] T 24d7b7f928ad446d8fe0d7cdc507c620 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting consensus metadata
I20240501 08:12:05.668895 31032 tablet_replica.cc:330] T 251e073eb0e44b02bb4ef9cb98aded4b P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:05.668920 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet 24d7b7f928ad446d8fe0d7cdc507c620 (table rebalance_test_table_4 [id=9c7af050a91047f384bcfb6909198411]) successfully deleted
I20240501 08:12:05.669342 31032 raft_consensus.cc:2229] T 251e073eb0e44b02bb4ef9cb98aded4b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:05.669682 31032 raft_consensus.cc:2258] T 251e073eb0e44b02bb4ef9cb98aded4b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:05.670500 31032 ts_tablet_manager.cc:1875] T 251e073eb0e44b02bb4ef9cb98aded4b P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:05.681922 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "928b70089c234425862c033341c82bad" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:12:05.682235 31160 tablet_bootstrap.cc:492] T 928b70089c234425862c033341c82bad P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:12:05.682425 31160 ts_tablet_manager.cc:1367] T 928b70089c234425862c033341c82bad P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.090s user 0.009s sys 0.016s
I20240501 08:12:05.682710 31166 raft_consensus.cc:348] T 683aa9595f2e456c943fc4d16f22e83c P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
W20240501 08:12:05.682802 30740 leader_election.cc:341] T 928b70089c234425862c033341c82bad P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:05.682936 31166 raft_consensus.cc:374] T 683aa9595f2e456c943fc4d16f22e83c P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:05.683050 31166 raft_consensus.cc:729] T 683aa9595f2e456c943fc4d16f22e83c P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:05.683336 31160 raft_consensus.cc:348] T 928b70089c234425862c033341c82bad P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:05.683336 31166 consensus_queue.cc:260] T 683aa9595f2e456c943fc4d16f22e83c P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:05.683517 31160 raft_consensus.cc:374] T 928b70089c234425862c033341c82bad P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:05.683599 31160 raft_consensus.cc:729] T 928b70089c234425862c033341c82bad P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:12:05.683849 31160 consensus_queue.cc:260] T 928b70089c234425862c033341c82bad P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:05.684471 31160 ts_tablet_manager.cc:1398] T 928b70089c234425862c033341c82bad P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:12:05.684767 31160 tablet_bootstrap.cc:492] T bc84d1d413994a76bd6cf42335438755 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:05.686151 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "928b70089c234425862c033341c82bad" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
W20240501 08:12:05.687192 30737 leader_election.cc:341] T 928b70089c234425862c033341c82bad P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:05.687361 30737 leader_election.cc:302] T 928b70089c234425862c033341c82bad P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5; no voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af
I20240501 08:12:05.687403 31158 ts_tablet_manager.cc:1888] T 24d7b7f928ad446d8fe0d7cdc507c620 P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:05.687549 31158 log.cc:1196] T 24d7b7f928ad446d8fe0d7cdc507c620 P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/24d7b7f928ad446d8fe0d7cdc507c620
I20240501 08:12:05.688042 31158 ts_tablet_manager.cc:1909] T 24d7b7f928ad446d8fe0d7cdc507c620 P aad81047009a4ffcbc450b60030583b5: Deleting consensus metadata
I20240501 08:12:05.688621 31082 raft_consensus.cc:2726] T 928b70089c234425862c033341c82bad P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:05.689285 31163 tablet_bootstrap.cc:492] T bc84d1d413994a76bd6cf42335438755 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:05.689396 31160 tablet_bootstrap.cc:654] T bc84d1d413994a76bd6cf42335438755 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:05.689553 31163 ts_tablet_manager.cc:1367] T bc84d1d413994a76bd6cf42335438755 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.000s
I20240501 08:12:05.690326 31161 tablet_bootstrap.cc:654] T 683aa9595f2e456c943fc4d16f22e83c P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:05.690565 31163 raft_consensus.cc:348] T bc84d1d413994a76bd6cf42335438755 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:05.690877 31163 raft_consensus.cc:374] T bc84d1d413994a76bd6cf42335438755 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:05.691015 31163 raft_consensus.cc:729] T bc84d1d413994a76bd6cf42335438755 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:05.691365 31163 consensus_queue.cc:260] T bc84d1d413994a76bd6cf42335438755 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:05.692118 31163 ts_tablet_manager.cc:1398] T bc84d1d413994a76bd6cf42335438755 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:05.718314 31165 tablet_bootstrap.cc:492] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:05.721978 31165 ts_tablet_manager.cc:1367] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.099s user 0.006s sys 0.000s
I20240501 08:12:05.723171 31165 raft_consensus.cc:348] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:05.723587 31165 raft_consensus.cc:374] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:05.723748 31165 raft_consensus.cc:729] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:05.724893 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet 24d7b7f928ad446d8fe0d7cdc507c620 (table rebalance_test_table_4 [id=9c7af050a91047f384bcfb6909198411]) successfully deleted
I20240501 08:12:05.724866 31165 consensus_queue.cc:260] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:05.726269 31158 tablet_replica.cc:330] T 251e073eb0e44b02bb4ef9cb98aded4b P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:05.726574 31158 raft_consensus.cc:2229] T 251e073eb0e44b02bb4ef9cb98aded4b P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:05.726730 31166 ts_tablet_manager.cc:1398] T 683aa9595f2e456c943fc4d16f22e83c P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.062s user 0.001s sys 0.000s
I20240501 08:12:05.729130 31161 tablet_bootstrap.cc:492] T 683aa9595f2e456c943fc4d16f22e83c P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:05.729710 31158 raft_consensus.cc:2258] T 251e073eb0e44b02bb4ef9cb98aded4b P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:05.729403 31161 ts_tablet_manager.cc:1367] T 683aa9595f2e456c943fc4d16f22e83c P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.085s user 0.000s sys 0.004s
I20240501 08:12:05.732096 31161 raft_consensus.cc:348] T 683aa9595f2e456c943fc4d16f22e83c P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:05.732407 31161 raft_consensus.cc:374] T 683aa9595f2e456c943fc4d16f22e83c P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:05.732525 31161 raft_consensus.cc:729] T 683aa9595f2e456c943fc4d16f22e83c P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:05.732952 31161 consensus_queue.cc:260] T 683aa9595f2e456c943fc4d16f22e83c P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:05.734565 31161 ts_tablet_manager.cc:1398] T 683aa9595f2e456c943fc4d16f22e83c P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:12:05.734961 31161 tablet_bootstrap.cc:492] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:05.736331 31158 ts_tablet_manager.cc:1875] T 251e073eb0e44b02bb4ef9cb98aded4b P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:05.737641 31161 tablet_bootstrap.cc:654] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:05.751935 31161 tablet_bootstrap.cc:492] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:05.752211 31161 ts_tablet_manager.cc:1367] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.004s
I20240501 08:12:05.752821 31164 tablet_bootstrap.cc:492] T 928b70089c234425862c033341c82bad P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:12:05.753080 31164 ts_tablet_manager.cc:1367] T 928b70089c234425862c033341c82bad P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.134s user 0.002s sys 0.004s
I20240501 08:12:05.754261 31161 raft_consensus.cc:348] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:05.754559 31161 raft_consensus.cc:374] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:05.754688 31161 raft_consensus.cc:729] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:05.755003 31161 consensus_queue.cc:260] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:05.755199 31164 raft_consensus.cc:348] T 928b70089c234425862c033341c82bad P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:05.755580 31164 raft_consensus.cc:374] T 928b70089c234425862c033341c82bad P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:05.755741 31164 raft_consensus.cc:729] T 928b70089c234425862c033341c82bad P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:12:05.756093 31164 consensus_queue.cc:260] T 928b70089c234425862c033341c82bad P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:05.756680 31161 ts_tablet_manager.cc:1398] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:05.756985 27618 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:05.772476 31164 ts_tablet_manager.cc:1398] T 928b70089c234425862c033341c82bad P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.019s user 0.000s sys 0.002s
I20240501 08:12:05.777395 31151 tablet.cc:1722] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:05.778378 31153 tablet.cc:1722] T 1abd3e69215d41a78147d9997ba48a3c P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:05.784797 31158 ts_tablet_manager.cc:1888] T 251e073eb0e44b02bb4ef9cb98aded4b P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:05.785060 31158 log.cc:1196] T 251e073eb0e44b02bb4ef9cb98aded4b P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/251e073eb0e44b02bb4ef9cb98aded4b
I20240501 08:12:05.785887 31158 ts_tablet_manager.cc:1909] T 251e073eb0e44b02bb4ef9cb98aded4b P aad81047009a4ffcbc450b60030583b5: Deleting consensus metadata
I20240501 08:12:05.787814 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet 251e073eb0e44b02bb4ef9cb98aded4b (table rebalance_test_table_4 [id=9c7af050a91047f384bcfb6909198411]) successfully deleted
I20240501 08:12:05.789402 27620 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table_new_name [id=033cf002db9946f69084e909cf5e8a48] alter complete (version 3)
I20240501 08:12:05.794092 31165 ts_tablet_manager.cc:1398] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.072s user 0.003s sys 0.000s
I20240501 08:12:05.799319 31160 tablet_bootstrap.cc:492] T bc84d1d413994a76bd6cf42335438755 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:12:05.799988 31160 ts_tablet_manager.cc:1367] T bc84d1d413994a76bd6cf42335438755 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.115s user 0.021s sys 0.030s
I20240501 08:12:05.801811 31152 tablet.cc:1722] T 1abd3e69215d41a78147d9997ba48a3c P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:05.802649 31032 ts_tablet_manager.cc:1888] T 251e073eb0e44b02bb4ef9cb98aded4b P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:05.802822 31032 log.cc:1196] T 251e073eb0e44b02bb4ef9cb98aded4b P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/251e073eb0e44b02bb4ef9cb98aded4b
I20240501 08:12:05.803212 31032 ts_tablet_manager.cc:1909] T 251e073eb0e44b02bb4ef9cb98aded4b P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting consensus metadata
I20240501 08:12:05.804208 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet 251e073eb0e44b02bb4ef9cb98aded4b (table rebalance_test_table_4 [id=9c7af050a91047f384bcfb6909198411]) successfully deleted
I20240501 08:12:05.801898 31160 raft_consensus.cc:348] T bc84d1d413994a76bd6cf42335438755 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:05.813007 31160 raft_consensus.cc:374] T bc84d1d413994a76bd6cf42335438755 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:05.813123 31160 raft_consensus.cc:729] T bc84d1d413994a76bd6cf42335438755 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:12:05.813381 31160 consensus_queue.cc:260] T bc84d1d413994a76bd6cf42335438755 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:05.814049 31160 ts_tablet_manager.cc:1398] T bc84d1d413994a76bd6cf42335438755 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20240501 08:12:05.826756 31011 raft_consensus.cc:482] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:05.826992 31011 raft_consensus.cc:504] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:05.827675 31011 leader_election.cc:288] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:05.828599 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "683aa9595f2e456c943fc4d16f22e83c" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:05.828881 30803 raft_consensus.cc:2454] T 683aa9595f2e456c943fc4d16f22e83c P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 0.
I20240501 08:12:05.829303 30937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "683aa9595f2e456c943fc4d16f22e83c" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:05.829573 30937 raft_consensus.cc:2454] T 683aa9595f2e456c943fc4d16f22e83c P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 0.
I20240501 08:12:05.830032 30606 leader_election.cc:302] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:05.830327 31011 raft_consensus.cc:2781] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:05.830476 31011 raft_consensus.cc:482] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:05.830566 31011 raft_consensus.cc:3037] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:05.833473 31011 raft_consensus.cc:504] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:05.838734 31011 leader_election.cc:288] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 election: Requested vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:05.838771 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "683aa9595f2e456c943fc4d16f22e83c" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:05.839037 30803 raft_consensus.cc:3037] T 683aa9595f2e456c943fc4d16f22e83c P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:05.842276 30937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "683aa9595f2e456c943fc4d16f22e83c" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:05.842511 30937 raft_consensus.cc:3037] T 683aa9595f2e456c943fc4d16f22e83c P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:05.842551 30803 raft_consensus.cc:2454] T 683aa9595f2e456c943fc4d16f22e83c P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 1.
I20240501 08:12:05.843102 30606 leader_election.cc:302] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:05.843425 31011 raft_consensus.cc:2781] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:05.844146 31011 raft_consensus.cc:686] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Becoming Leader. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Running, Role: LEADER
I20240501 08:12:05.844615 31011 consensus_queue.cc:237] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:05.845314 30937 raft_consensus.cc:2454] T 683aa9595f2e456c943fc4d16f22e83c P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 1.
I20240501 08:12:05.852012 27620 catalog_manager.cc:5484] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134). New cstate: current_term: 1 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:05.892223 27620 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:05.892482 27620 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:05.896117 27620 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 1abd3e69215d41a78147d9997ba48a3c
I20240501 08:12:05.897095 30652 tablet_service.cc:1514] Processing DeleteTablet for tablet 1abd3e69215d41a78147d9997ba48a3c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:05 UTC) from {username='slave'} at 127.0.0.1:57156
I20240501 08:12:05.897306 27739 tablet_service.cc:1514] Processing DeleteTablet for tablet 1abd3e69215d41a78147d9997ba48a3c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:05 UTC) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:05.897481 30517 tablet_service.cc:1514] Processing DeleteTablet for tablet 1abd3e69215d41a78147d9997ba48a3c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:05 UTC) from {username='slave'} at 127.0.0.1:45078
I20240501 08:12:05.897876 31032 tablet_replica.cc:330] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:05.898226 31032 raft_consensus.cc:2229] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:05.898564 31032 raft_consensus.cc:2258] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:05.899987 31032 ts_tablet_manager.cc:1875] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:05.904560 31198 tablet_replica.cc:330] T 1abd3e69215d41a78147d9997ba48a3c P 31811708c6774037b82df44024b229b6: stopping tablet replica
I20240501 08:12:05.905256 31198 raft_consensus.cc:2229] T 1abd3e69215d41a78147d9997ba48a3c P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:05.905652 31198 raft_consensus.cc:2258] T 1abd3e69215d41a78147d9997ba48a3c P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:05.907153 31198 ts_tablet_manager.cc:1875] T 1abd3e69215d41a78147d9997ba48a3c P 31811708c6774037b82df44024b229b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:05.909502 31032 ts_tablet_manager.cc:1888] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:05.909667 31032 log.cc:1196] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/1abd3e69215d41a78147d9997ba48a3c
I20240501 08:12:05.910140 31032 ts_tablet_manager.cc:1909] T 1abd3e69215d41a78147d9997ba48a3c P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting consensus metadata
I20240501 08:12:05.913724 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet 1abd3e69215d41a78147d9997ba48a3c (table rebalancer_dynamic_table_new_name [id=033cf002db9946f69084e909cf5e8a48]) successfully deleted
I20240501 08:12:05.918506 31198 ts_tablet_manager.cc:1888] T 1abd3e69215d41a78147d9997ba48a3c P 31811708c6774037b82df44024b229b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:05.925617 31197 tablet_replica.cc:330] T 1abd3e69215d41a78147d9997ba48a3c P 9a022e1a501f4d44a0468054d6c3de47: stopping tablet replica
I20240501 08:12:05.925603 31198 log.cc:1196] T 1abd3e69215d41a78147d9997ba48a3c P 31811708c6774037b82df44024b229b6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/wals/1abd3e69215d41a78147d9997ba48a3c
I20240501 08:12:05.925982 31197 raft_consensus.cc:2229] T 1abd3e69215d41a78147d9997ba48a3c P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:05.926220 31198 ts_tablet_manager.cc:1909] T 1abd3e69215d41a78147d9997ba48a3c P 31811708c6774037b82df44024b229b6: Deleting consensus metadata
I20240501 08:12:05.926259 31197 raft_consensus.cc:2258] T 1abd3e69215d41a78147d9997ba48a3c P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:05.927253 31197 ts_tablet_manager.cc:1875] T 1abd3e69215d41a78147d9997ba48a3c P 9a022e1a501f4d44a0468054d6c3de47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:05.927347 27607 catalog_manager.cc:4830] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): tablet 1abd3e69215d41a78147d9997ba48a3c (table rebalancer_dynamic_table_new_name [id=033cf002db9946f69084e909cf5e8a48]) successfully deleted
I20240501 08:12:05.935004 31197 ts_tablet_manager.cc:1888] T 1abd3e69215d41a78147d9997ba48a3c P 9a022e1a501f4d44a0468054d6c3de47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:05.935277 31197 log.cc:1196] T 1abd3e69215d41a78147d9997ba48a3c P 9a022e1a501f4d44a0468054d6c3de47: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/wals/1abd3e69215d41a78147d9997ba48a3c
I20240501 08:12:05.936046 31197 ts_tablet_manager.cc:1909] T 1abd3e69215d41a78147d9997ba48a3c P 9a022e1a501f4d44a0468054d6c3de47: Deleting consensus metadata
I20240501 08:12:05.937664 27604 catalog_manager.cc:4830] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): tablet 1abd3e69215d41a78147d9997ba48a3c (table rebalancer_dynamic_table_new_name [id=033cf002db9946f69084e909cf5e8a48]) successfully deleted
I20240501 08:12:05.939569 31082 raft_consensus.cc:482] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:05.939733 31082 raft_consensus.cc:504] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:05.940488 31082 leader_election.cc:288] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:05.942267 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc84d1d413994a76bd6cf42335438755" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:12:05.942514 27890 raft_consensus.cc:2454] T bc84d1d413994a76bd6cf42335438755 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:05.943392 30740 leader_election.cc:302] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:05.943742 31082 raft_consensus.cc:2781] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:05.943854 31082 raft_consensus.cc:482] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:05.943933 31082 raft_consensus.cc:3037] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:05.949371 31082 raft_consensus.cc:504] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:05.951437 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc84d1d413994a76bd6cf42335438755" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:12:05.951727 27890 raft_consensus.cc:3037] T bc84d1d413994a76bd6cf42335438755 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:05.952109 31082 leader_election.cc:288] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Requested vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:05.954830 27890 raft_consensus.cc:2454] T bc84d1d413994a76bd6cf42335438755 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:05.955484 30740 leader_election.cc:302] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:05.955947 31082 raft_consensus.cc:2781] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:05.956521 31082 raft_consensus.cc:686] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Becoming Leader. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Running, Role: LEADER
I20240501 08:12:05.957187 31082 consensus_queue.cc:237] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:05.965813 27620 catalog_manager.cc:5484] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64998323a1f4482d8704bf20594d13a5 (127.21.84.135). New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:05.969388 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc84d1d413994a76bd6cf42335438755" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:05.969488 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc84d1d413994a76bd6cf42335438755" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:05.969650 30536 raft_consensus.cc:2454] T bc84d1d413994a76bd6cf42335438755 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:05.986605 31056 raft_consensus.cc:482] T 928b70089c234425862c033341c82bad P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:05.986795 31056 raft_consensus.cc:504] T 928b70089c234425862c033341c82bad P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:05.987699 31056 leader_election.cc:288] T 928b70089c234425862c033341c82bad P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:05.988961 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "928b70089c234425862c033341c82bad" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:12:05.989269 28150 raft_consensus.cc:2454] T 928b70089c234425862c033341c82bad P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:12:05.989856 27825 leader_election.cc:302] T 928b70089c234425862c033341c82bad P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:05.990319 31056 raft_consensus.cc:2781] T 928b70089c234425862c033341c82bad P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:05.990455 31056 raft_consensus.cc:482] T 928b70089c234425862c033341c82bad P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:05.990551 31056 raft_consensus.cc:3037] T 928b70089c234425862c033341c82bad P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:05.996976 31056 raft_consensus.cc:504] T 928b70089c234425862c033341c82bad P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:05.998198 31056 leader_election.cc:288] T 928b70089c234425862c033341c82bad P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Requested vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:05.998490 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "928b70089c234425862c033341c82bad" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:12:05.998766 28150 raft_consensus.cc:3037] T 928b70089c234425862c033341c82bad P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:06.004410 27613 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240501 08:12:06.004510 27620 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:06.004771 27613 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240501 08:12:06.009687 28150 raft_consensus.cc:2454] T 928b70089c234425862c033341c82bad P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:12:06.010469 27825 leader_election.cc:302] T 928b70089c234425862c033341c82bad P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:06.011121 31056 raft_consensus.cc:2781] T 928b70089c234425862c033341c82bad P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:06.011289 31056 raft_consensus.cc:686] T 928b70089c234425862c033341c82bad P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Becoming Leader. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Running, Role: LEADER
I20240501 08:12:06.011678 31056 consensus_queue.cc:237] T 928b70089c234425862c033341c82bad P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:06.012015 27613 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 20425913301a4d8abb9621cc7db0e3e7
I20240501 08:12:06.013581 27872 tablet_service.cc:1514] Processing DeleteTablet for tablet 20425913301a4d8abb9621cc7db0e3e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:05 UTC) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:06.013674 28000 tablet_service.cc:1514] Processing DeleteTablet for tablet 20425913301a4d8abb9621cc7db0e3e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:05 UTC) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:06.014001 31158 tablet_replica.cc:330] T 20425913301a4d8abb9621cc7db0e3e7 P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:06.014281 31158 raft_consensus.cc:2229] T 20425913301a4d8abb9621cc7db0e3e7 P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:06.014876 31158 raft_consensus.cc:2258] T 20425913301a4d8abb9621cc7db0e3e7 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:06.015764 31158 ts_tablet_manager.cc:1875] T 20425913301a4d8abb9621cc7db0e3e7 P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:06.017139 30652 tablet_service.cc:1467] Processing CreateTablet for tablet 95732c57c31a4f899dab7f60c1c89e48 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c3119cae82434187b5457d0f9c9309e8]), partition=
I20240501 08:12:06.017163 30783 tablet_service.cc:1467] Processing CreateTablet for tablet 95732c57c31a4f899dab7f60c1c89e48 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c3119cae82434187b5457d0f9c9309e8]), partition=
I20240501 08:12:06.017503 30783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95732c57c31a4f899dab7f60c1c89e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:06.017530 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95732c57c31a4f899dab7f60c1c89e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:06.021258 27739 tablet_service.cc:1514] Processing DeleteTablet for tablet 20425913301a4d8abb9621cc7db0e3e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:05 UTC) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:06.021459 27613 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet e258bb4e672f4795a3b6739f8636716e
I20240501 08:12:06.021567 31032 tablet_replica.cc:330] T 20425913301a4d8abb9621cc7db0e3e7 P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:06.021771 31032 raft_consensus.cc:2229] T 20425913301a4d8abb9621cc7db0e3e7 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:06.022065 31032 raft_consensus.cc:2258] T 20425913301a4d8abb9621cc7db0e3e7 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:06.022866 31032 ts_tablet_manager.cc:1875] T 20425913301a4d8abb9621cc7db0e3e7 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:06.025568 27613 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 0d9538e4316a403e805ba5d06e6585d8
I20240501 08:12:06.026368 28132 tablet_service.cc:1467] Processing CreateTablet for tablet 95732c57c31a4f899dab7f60c1c89e48 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c3119cae82434187b5457d0f9c9309e8]), partition=
I20240501 08:12:06.026738 28132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95732c57c31a4f899dab7f60c1c89e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:06.027447 28000 tablet_service.cc:1514] Processing DeleteTablet for tablet e258bb4e672f4795a3b6739f8636716e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:05 UTC) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:06.029103 31161 tablet_bootstrap.cc:492] T 95732c57c31a4f899dab7f60c1c89e48 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:06.030077 27741 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d9538e4316a403e805ba5d06e6585d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:05 UTC) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:06.030105 28000 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d9538e4316a403e805ba5d06e6585d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:05 UTC) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:06.030547 28130 tablet_service.cc:1514] Processing DeleteTablet for tablet e258bb4e672f4795a3b6739f8636716e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:05 UTC) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:06.030704 31161 tablet_bootstrap.cc:654] T 95732c57c31a4f899dab7f60c1c89e48 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:06.030835 31031 tablet_replica.cc:330] T e258bb4e672f4795a3b6739f8636716e P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:06.031075 31031 raft_consensus.cc:2229] T e258bb4e672f4795a3b6739f8636716e P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:06.031385 31031 raft_consensus.cc:2258] T e258bb4e672f4795a3b6739f8636716e P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:06.032370 31031 ts_tablet_manager.cc:1875] T e258bb4e672f4795a3b6739f8636716e P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:06.034919 27739 tablet_service.cc:1514] Processing DeleteTablet for tablet e258bb4e672f4795a3b6739f8636716e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:05 UTC) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:06.037741 27870 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d9538e4316a403e805ba5d06e6585d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:05 UTC) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:06.049696 31161 tablet_bootstrap.cc:492] T 95732c57c31a4f899dab7f60c1c89e48 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:06.049928 31161 ts_tablet_manager.cc:1367] T 95732c57c31a4f899dab7f60c1c89e48 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.000s
I20240501 08:12:06.050799 31161 raft_consensus.cc:348] T 95732c57c31a4f899dab7f60c1c89e48 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:06.051044 31161 raft_consensus.cc:374] T 95732c57c31a4f899dab7f60c1c89e48 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:06.051126 31161 raft_consensus.cc:729] T 95732c57c31a4f899dab7f60c1c89e48 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:06.051393 31161 consensus_queue.cc:260] T 95732c57c31a4f899dab7f60c1c89e48 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:06.052647 31161 ts_tablet_manager.cc:1398] T 95732c57c31a4f899dab7f60c1c89e48 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:12:06.055686 31165 tablet_bootstrap.cc:492] T 95732c57c31a4f899dab7f60c1c89e48 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:06.057354 31165 tablet_bootstrap.cc:654] T 95732c57c31a4f899dab7f60c1c89e48 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:06.061322 31164 tablet_bootstrap.cc:492] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:06.063103 31164 tablet_bootstrap.cc:654] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:06.065439 31165 tablet_bootstrap.cc:492] T 95732c57c31a4f899dab7f60c1c89e48 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:06.065711 31165 ts_tablet_manager.cc:1367] T 95732c57c31a4f899dab7f60c1c89e48 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.004s
I20240501 08:12:06.050678 27613 catalog_manager.cc:5484] T 928b70089c234425862c033341c82bad P f1525cbc82a24d13b65f136695c903af reported cstate change: term changed from 0 to 1, leader changed from <none> to f1525cbc82a24d13b65f136695c903af (127.21.84.130). New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:06.067037 31165 raft_consensus.cc:348] T 95732c57c31a4f899dab7f60c1c89e48 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:06.067543 31165 raft_consensus.cc:374] T 95732c57c31a4f899dab7f60c1c89e48 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:06.067772 31165 raft_consensus.cc:729] T 95732c57c31a4f899dab7f60c1c89e48 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:06.068292 31165 consensus_queue.cc:260] T 95732c57c31a4f899dab7f60c1c89e48 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:06.070219 31165 ts_tablet_manager.cc:1398] T 95732c57c31a4f899dab7f60c1c89e48 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 08:12:06.076172 31031 ts_tablet_manager.cc:1888] T e258bb4e672f4795a3b6739f8636716e P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:06.077433 31032 ts_tablet_manager.cc:1888] T 20425913301a4d8abb9621cc7db0e3e7 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:06.078559 31032 log.cc:1196] T 20425913301a4d8abb9621cc7db0e3e7 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/20425913301a4d8abb9621cc7db0e3e7
I20240501 08:12:06.083994 31032 ts_tablet_manager.cc:1909] T 20425913301a4d8abb9621cc7db0e3e7 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting consensus metadata
I20240501 08:12:06.085301 31031 log.cc:1196] T e258bb4e672f4795a3b6739f8636716e P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/e258bb4e672f4795a3b6739f8636716e
I20240501 08:12:06.087625 31158 ts_tablet_manager.cc:1888] T 20425913301a4d8abb9621cc7db0e3e7 P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:06.087780 31031 ts_tablet_manager.cc:1909] T e258bb4e672f4795a3b6739f8636716e P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting consensus metadata
I20240501 08:12:06.089612 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet 20425913301a4d8abb9621cc7db0e3e7 (table rebalance_test_table_6 [id=bb586b867d1742378689aab16d2baa26]) successfully deleted
I20240501 08:12:06.089946 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet e258bb4e672f4795a3b6739f8636716e (table rebalance_test_table_6 [id=bb586b867d1742378689aab16d2baa26]) successfully deleted
I20240501 08:12:06.091053 31032 tablet_replica.cc:330] T 0d9538e4316a403e805ba5d06e6585d8 P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:06.091300 31032 raft_consensus.cc:2229] T 0d9538e4316a403e805ba5d06e6585d8 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:06.091634 31032 raft_consensus.cc:2258] T 0d9538e4316a403e805ba5d06e6585d8 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:06.094056 31032 ts_tablet_manager.cc:1875] T 0d9538e4316a403e805ba5d06e6585d8 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:06.096118 31164 tablet_bootstrap.cc:492] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:12:06.096367 31164 ts_tablet_manager.cc:1367] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.035s user 0.000s sys 0.009s
I20240501 08:12:06.096455 31223 tablet_replica.cc:330] T 20425913301a4d8abb9621cc7db0e3e7 P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:06.096869 31223 raft_consensus.cc:2229] T 20425913301a4d8abb9621cc7db0e3e7 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:06.097142 31158 log.cc:1196] T 20425913301a4d8abb9621cc7db0e3e7 P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/20425913301a4d8abb9621cc7db0e3e7
I20240501 08:12:06.097234 31164 raft_consensus.cc:348] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:06.097530 31164 raft_consensus.cc:374] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:06.097645 31164 raft_consensus.cc:729] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:12:06.097921 31158 ts_tablet_manager.cc:1909] T 20425913301a4d8abb9621cc7db0e3e7 P aad81047009a4ffcbc450b60030583b5: Deleting consensus metadata
I20240501 08:12:06.097996 31164 consensus_queue.cc:260] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:06.098814 31164 ts_tablet_manager.cc:1398] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:06.102810 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "928b70089c234425862c033341c82bad" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:06.102897 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "928b70089c234425862c033341c82bad" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:06.103080 30803 raft_consensus.cc:2454] T 928b70089c234425862c033341c82bad P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:12:06.105707 31223 raft_consensus.cc:2258] T 20425913301a4d8abb9621cc7db0e3e7 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:06.106734 31223 ts_tablet_manager.cc:1875] T 20425913301a4d8abb9621cc7db0e3e7 P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:06.107406 31032 ts_tablet_manager.cc:1888] T 0d9538e4316a403e805ba5d06e6585d8 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:06.107565 31032 log.cc:1196] T 0d9538e4316a403e805ba5d06e6585d8 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/0d9538e4316a403e805ba5d06e6585d8
I20240501 08:12:06.108101 31032 ts_tablet_manager.cc:1909] T 0d9538e4316a403e805ba5d06e6585d8 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting consensus metadata
I20240501 08:12:06.109252 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet 0d9538e4316a403e805ba5d06e6585d8 (table rebalance_test_table_6 [id=bb586b867d1742378689aab16d2baa26]) successfully deleted
I20240501 08:12:06.111003 31032 tablet_replica.cc:330] T e258bb4e672f4795a3b6739f8636716e P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:06.110999 31158 tablet_replica.cc:330] T e258bb4e672f4795a3b6739f8636716e P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:06.111155 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet 20425913301a4d8abb9621cc7db0e3e7 (table rebalance_test_table_6 [id=bb586b867d1742378689aab16d2baa26]) successfully deleted
I20240501 08:12:06.111202 31032 raft_consensus.cc:2229] T e258bb4e672f4795a3b6739f8636716e P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:06.111507 31032 raft_consensus.cc:2258] T e258bb4e672f4795a3b6739f8636716e P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:06.112250 31032 ts_tablet_manager.cc:1875] T e258bb4e672f4795a3b6739f8636716e P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:06.113310 31158 raft_consensus.cc:2229] T e258bb4e672f4795a3b6739f8636716e P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:06.113582 31158 raft_consensus.cc:2258] T e258bb4e672f4795a3b6739f8636716e P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:06.118739 31032 ts_tablet_manager.cc:1888] T e258bb4e672f4795a3b6739f8636716e P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:06.118902 31032 log.cc:1196] T e258bb4e672f4795a3b6739f8636716e P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/e258bb4e672f4795a3b6739f8636716e
I20240501 08:12:06.119421 31032 ts_tablet_manager.cc:1909] T e258bb4e672f4795a3b6739f8636716e P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting consensus metadata
I20240501 08:12:06.119880 31158 ts_tablet_manager.cc:1875] T e258bb4e672f4795a3b6739f8636716e P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:06.120676 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet e258bb4e672f4795a3b6739f8636716e (table rebalance_test_table_6 [id=bb586b867d1742378689aab16d2baa26]) successfully deleted
I20240501 08:12:06.126596 31158 ts_tablet_manager.cc:1888] T e258bb4e672f4795a3b6739f8636716e P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:06.126832 31158 log.cc:1196] T e258bb4e672f4795a3b6739f8636716e P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/e258bb4e672f4795a3b6739f8636716e
I20240501 08:12:06.127811 31158 ts_tablet_manager.cc:1909] T e258bb4e672f4795a3b6739f8636716e P aad81047009a4ffcbc450b60030583b5: Deleting consensus metadata
I20240501 08:12:06.128970 31158 tablet_replica.cc:330] T 0d9538e4316a403e805ba5d06e6585d8 P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:06.129089 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet e258bb4e672f4795a3b6739f8636716e (table rebalance_test_table_6 [id=bb586b867d1742378689aab16d2baa26]) successfully deleted
I20240501 08:12:06.129343 31158 raft_consensus.cc:2229] T 0d9538e4316a403e805ba5d06e6585d8 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:06.129638 31158 raft_consensus.cc:2258] T 0d9538e4316a403e805ba5d06e6585d8 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:06.130393 31158 ts_tablet_manager.cc:1875] T 0d9538e4316a403e805ba5d06e6585d8 P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:06.130745 31223 ts_tablet_manager.cc:1888] T 20425913301a4d8abb9621cc7db0e3e7 P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:06.130980 31223 log.cc:1196] T 20425913301a4d8abb9621cc7db0e3e7 P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/20425913301a4d8abb9621cc7db0e3e7
I20240501 08:12:06.134042 31223 ts_tablet_manager.cc:1909] T 20425913301a4d8abb9621cc7db0e3e7 P f1525cbc82a24d13b65f136695c903af: Deleting consensus metadata
I20240501 08:12:06.137198 31158 ts_tablet_manager.cc:1888] T 0d9538e4316a403e805ba5d06e6585d8 P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:06.137408 31158 log.cc:1196] T 0d9538e4316a403e805ba5d06e6585d8 P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/0d9538e4316a403e805ba5d06e6585d8
I20240501 08:12:06.138033 31158 ts_tablet_manager.cc:1909] T 0d9538e4316a403e805ba5d06e6585d8 P aad81047009a4ffcbc450b60030583b5: Deleting consensus metadata
I20240501 08:12:06.139694 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet 0d9538e4316a403e805ba5d06e6585d8 (table rebalance_test_table_6 [id=bb586b867d1742378689aab16d2baa26]) successfully deleted
I20240501 08:12:06.146044 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet 20425913301a4d8abb9621cc7db0e3e7 (table rebalance_test_table_6 [id=bb586b867d1742378689aab16d2baa26]) successfully deleted
I20240501 08:12:06.146508 31223 tablet_replica.cc:330] T 0d9538e4316a403e805ba5d06e6585d8 P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:06.146948 31223 raft_consensus.cc:2229] T 0d9538e4316a403e805ba5d06e6585d8 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:06.147328 31223 raft_consensus.cc:2258] T 0d9538e4316a403e805ba5d06e6585d8 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:06.148429 31223 ts_tablet_manager.cc:1875] T 0d9538e4316a403e805ba5d06e6585d8 P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:06.162829 31223 ts_tablet_manager.cc:1888] T 0d9538e4316a403e805ba5d06e6585d8 P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:06.163146 31223 log.cc:1196] T 0d9538e4316a403e805ba5d06e6585d8 P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/0d9538e4316a403e805ba5d06e6585d8
I20240501 08:12:06.163729 31223 ts_tablet_manager.cc:1909] T 0d9538e4316a403e805ba5d06e6585d8 P f1525cbc82a24d13b65f136695c903af: Deleting consensus metadata
I20240501 08:12:06.164886 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet 0d9538e4316a403e805ba5d06e6585d8 (table rebalance_test_table_6 [id=bb586b867d1742378689aab16d2baa26]) successfully deleted
I20240501 08:12:06.169611 31138 raft_consensus.cc:482] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:06.169808 31138 raft_consensus.cc:504] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:06.170755 31138 leader_election.cc:288] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:06.190248 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95732c57c31a4f899dab7f60c1c89e48" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:06.190505 30804 raft_consensus.cc:2454] T 95732c57c31a4f899dab7f60c1c89e48 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:12:06.190977 28086 leader_election.cc:302] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:12:06.191298 31138 raft_consensus.cc:2781] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:06.191418 31138 raft_consensus.cc:482] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:06.191493 31138 raft_consensus.cc:3037] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:06.194345 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95732c57c31a4f899dab7f60c1c89e48" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:06.194587 30672 raft_consensus.cc:2454] T 95732c57c31a4f899dab7f60c1c89e48 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:12:06.195516 31138 raft_consensus.cc:504] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:06.196247 31138 leader_election.cc:288] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:06.196662 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95732c57c31a4f899dab7f60c1c89e48" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:06.196872 30804 raft_consensus.cc:3037] T 95732c57c31a4f899dab7f60c1c89e48 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:06.198345 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95732c57c31a4f899dab7f60c1c89e48" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:06.198557 30672 raft_consensus.cc:3037] T 95732c57c31a4f899dab7f60c1c89e48 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:06.199733 30804 raft_consensus.cc:2454] T 95732c57c31a4f899dab7f60c1c89e48 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:12:06.200326 28086 leader_election.cc:302] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:12:06.200640 31138 raft_consensus.cc:2781] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:06.200838 31138 raft_consensus.cc:686] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:12:06.200816 30672 raft_consensus.cc:2454] T 95732c57c31a4f899dab7f60c1c89e48 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:12:06.201126 31138 consensus_queue.cc:237] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:06.205121 27613 catalog_manager.cc:5484] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:06.247289 31011 consensus_queue.cc:1035] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:06.256543 31179 consensus_queue.cc:1035] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:06.333125 27613 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:06.340021 30803 raft_consensus.cc:1261] T 95732c57c31a4f899dab7f60c1c89e48 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:06.340090 30672 raft_consensus.cc:1261] T 95732c57c31a4f899dab7f60c1c89e48 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:06.340683 31010 consensus_queue.cc:1035] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:06.340900 31138 consensus_queue.cc:1035] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:06.344887 27613 catalog_manager.cc:5576] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933) does not have the latest schema for tablet 95732c57c31a4f899dab7f60c1c89e48 (table rebalancer_dynamic_table [id=c3119cae82434187b5457d0f9c9309e8]). Expected version 1 got 0
I20240501 08:12:06.346097 27620 catalog_manager.cc:5576] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503) does not have the latest schema for tablet 95732c57c31a4f899dab7f60c1c89e48 (table rebalancer_dynamic_table [id=c3119cae82434187b5457d0f9c9309e8]). Expected version 1 got 0
I20240501 08:12:06.346829 31238 mvcc.cc:205] Tried to move back new op lower bound from 7022801413482291200 to 7022801412922478592. Current Snapshot: MvccSnapshot[applied={T|T < 7022801413482291200}]
I20240501 08:12:06.347543 31241 tablet.cc:1722] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:06.348913 31241 tablet.cc:1620] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6: MemRowSet was empty: no flush needed.
I20240501 08:12:06.350557 27620 catalog_manager.cc:5576] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323) does not have the latest schema for tablet 95732c57c31a4f899dab7f60c1c89e48 (table rebalancer_dynamic_table [id=c3119cae82434187b5457d0f9c9309e8]). Expected version 1 got 0
I20240501 08:12:06.353956 31235 mvcc.cc:205] Tried to move back new op lower bound from 7022801413482291200 to 7022801412922478592. Current Snapshot: MvccSnapshot[applied={T|T < 7022801413482291200}]
I20240501 08:12:06.357518 27613 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=c3119cae82434187b5457d0f9c9309e8] alter complete (version 1)
I20240501 08:12:06.357720 31243 tablet.cc:1722] T 95732c57c31a4f899dab7f60c1c89e48 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:06.358530 31243 tablet.cc:1620] T 95732c57c31a4f899dab7f60c1c89e48 P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:06.359179 31242 tablet.cc:1722] T 95732c57c31a4f899dab7f60c1c89e48 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:06.360242 27620 catalog_manager.cc:5576] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933) does not have the latest schema for tablet 95732c57c31a4f899dab7f60c1c89e48 (table rebalancer_dynamic_table [id=c3119cae82434187b5457d0f9c9309e8]). Expected version 1 got 0
I20240501 08:12:06.362928 27613 catalog_manager.cc:5576] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503) does not have the latest schema for tablet 95732c57c31a4f899dab7f60c1c89e48 (table rebalancer_dynamic_table [id=c3119cae82434187b5457d0f9c9309e8]). Expected version 1 got 0
I20240501 08:12:06.370327 31242 tablet.cc:1620] T 95732c57c31a4f899dab7f60c1c89e48 P 64998323a1f4482d8704bf20594d13a5: MemRowSet was empty: no flush needed.
I20240501 08:12:06.406383 31082 consensus_queue.cc:1035] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:06.411465 30537 raft_consensus.cc:3037] T bc84d1d413994a76bd6cf42335438755 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:06.415434 31205 consensus_queue.cc:1035] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:06.440285 31222 consensus_queue.cc:1035] T 928b70089c234425862c033341c82bad P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:06.443821 30803 raft_consensus.cc:3037] T 928b70089c234425862c033341c82bad P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:06.447429 31056 consensus_queue.cc:1035] T 928b70089c234425862c033341c82bad P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:06.460557 27620 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:06.467185 31241 tablet.cc:1722] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:06.467666 31241 tablet.cc:1620] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6: MemRowSet was empty: no flush needed.
I20240501 08:12:06.468063 31243 tablet.cc:1722] T 95732c57c31a4f899dab7f60c1c89e48 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:06.468122 31242 tablet.cc:1722] T 95732c57c31a4f899dab7f60c1c89e48 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:06.468520 31243 tablet.cc:1620] T 95732c57c31a4f899dab7f60c1c89e48 P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:06.468575 31242 tablet.cc:1620] T 95732c57c31a4f899dab7f60c1c89e48 P 64998323a1f4482d8704bf20594d13a5: MemRowSet was empty: no flush needed.
I20240501 08:12:06.478029 27613 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=c3119cae82434187b5457d0f9c9309e8] alter complete (version 2)
I20240501 08:12:06.539973 27613 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240501 08:12:06.540225 27613 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240501 08:12:06.543462 27613 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 44f9bd2dc83a4533afc4cff8e93164e3
I20240501 08:12:06.544028 27613 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 3b24297ce5da4790877c5f5672a73d8c
I20240501 08:12:06.544277 28000 tablet_service.cc:1514] Processing DeleteTablet for tablet 44f9bd2dc83a4533afc4cff8e93164e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:06 UTC) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:06.544476 27739 tablet_service.cc:1514] Processing DeleteTablet for tablet 44f9bd2dc83a4533afc4cff8e93164e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:06 UTC) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:06.544672 27613 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet dfb81aba648d4099a949064b7f4442be
I20240501 08:12:06.544778 31032 tablet_replica.cc:330] T 44f9bd2dc83a4533afc4cff8e93164e3 P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:06.545053 31032 raft_consensus.cc:2229] T 44f9bd2dc83a4533afc4cff8e93164e3 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:06.545286 31032 raft_consensus.cc:2258] T 44f9bd2dc83a4533afc4cff8e93164e3 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:06.545818 28130 tablet_service.cc:1514] Processing DeleteTablet for tablet dfb81aba648d4099a949064b7f4442be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:06 UTC) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:06.546104 31158 tablet_replica.cc:330] T 44f9bd2dc83a4533afc4cff8e93164e3 P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:06.546139 31031 tablet_replica.cc:330] T dfb81aba648d4099a949064b7f4442be P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:06.546242 27741 tablet_service.cc:1514] Processing DeleteTablet for tablet dfb81aba648d4099a949064b7f4442be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:06 UTC) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:06.546341 31158 raft_consensus.cc:2229] T 44f9bd2dc83a4533afc4cff8e93164e3 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:06.546360 31031 raft_consensus.cc:2229] T dfb81aba648d4099a949064b7f4442be P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:06.546423 31032 ts_tablet_manager.cc:1875] T 44f9bd2dc83a4533afc4cff8e93164e3 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:06.546574 31158 raft_consensus.cc:2258] T 44f9bd2dc83a4533afc4cff8e93164e3 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:06.546564 27739 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b24297ce5da4790877c5f5672a73d8c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:06 UTC) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:06.546710 31031 raft_consensus.cc:2258] T dfb81aba648d4099a949064b7f4442be P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:06.547169 28000 tablet_service.cc:1514] Processing DeleteTablet for tablet dfb81aba648d4099a949064b7f4442be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:06 UTC) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:06.547281 31158 ts_tablet_manager.cc:1875] T 44f9bd2dc83a4533afc4cff8e93164e3 P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:06.547466 28132 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b24297ce5da4790877c5f5672a73d8c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:06 UTC) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:06.547441 31031 ts_tablet_manager.cc:1875] T dfb81aba648d4099a949064b7f4442be P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:06.547904 27872 tablet_service.cc:1514] Processing DeleteTablet for tablet 44f9bd2dc83a4533afc4cff8e93164e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:06 UTC) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:06.548146 27870 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b24297ce5da4790877c5f5672a73d8c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:06 UTC) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:06.552816 31223 tablet_replica.cc:330] T 44f9bd2dc83a4533afc4cff8e93164e3 P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:06.553059 31223 raft_consensus.cc:2229] T 44f9bd2dc83a4533afc4cff8e93164e3 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:06.553292 31032 ts_tablet_manager.cc:1888] T 44f9bd2dc83a4533afc4cff8e93164e3 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:06.553431 31032 log.cc:1196] T 44f9bd2dc83a4533afc4cff8e93164e3 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/44f9bd2dc83a4533afc4cff8e93164e3
I20240501 08:12:06.553426 31223 raft_consensus.cc:2258] T 44f9bd2dc83a4533afc4cff8e93164e3 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:06.553848 31032 ts_tablet_manager.cc:1909] T 44f9bd2dc83a4533afc4cff8e93164e3 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting consensus metadata
I20240501 08:12:06.554188 31031 ts_tablet_manager.cc:1888] T dfb81aba648d4099a949064b7f4442be P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:06.554234 31223 ts_tablet_manager.cc:1875] T 44f9bd2dc83a4533afc4cff8e93164e3 P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:06.554306 31031 log.cc:1196] T dfb81aba648d4099a949064b7f4442be P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/dfb81aba648d4099a949064b7f4442be
I20240501 08:12:06.554737 31031 ts_tablet_manager.cc:1909] T dfb81aba648d4099a949064b7f4442be P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting consensus metadata
I20240501 08:12:06.554888 31032 tablet_replica.cc:330] T dfb81aba648d4099a949064b7f4442be P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:06.555096 31032 raft_consensus.cc:2229] T dfb81aba648d4099a949064b7f4442be P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:06.555110 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet 44f9bd2dc83a4533afc4cff8e93164e3 (table rebalance_test_table_8 [id=ea84ae8db01148a6896438374a278971]) successfully deleted
I20240501 08:12:06.555353 31032 raft_consensus.cc:2258] T dfb81aba648d4099a949064b7f4442be P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:06.556032 31032 ts_tablet_manager.cc:1875] T dfb81aba648d4099a949064b7f4442be P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:06.556066 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet dfb81aba648d4099a949064b7f4442be (table rebalance_test_table_8 [id=ea84ae8db01148a6896438374a278971]) successfully deleted
I20240501 08:12:06.556424 31031 tablet_replica.cc:330] T 3b24297ce5da4790877c5f5672a73d8c P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:06.556720 31031 raft_consensus.cc:2229] T 3b24297ce5da4790877c5f5672a73d8c P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:06.557081 31031 raft_consensus.cc:2258] T 3b24297ce5da4790877c5f5672a73d8c P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:06.557886 31031 ts_tablet_manager.cc:1875] T 3b24297ce5da4790877c5f5672a73d8c P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:06.562665 31032 ts_tablet_manager.cc:1888] T dfb81aba648d4099a949064b7f4442be P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:06.562857 31032 log.cc:1196] T dfb81aba648d4099a949064b7f4442be P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/dfb81aba648d4099a949064b7f4442be
I20240501 08:12:06.563345 31032 ts_tablet_manager.cc:1909] T dfb81aba648d4099a949064b7f4442be P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting consensus metadata
I20240501 08:12:06.564399 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet dfb81aba648d4099a949064b7f4442be (table rebalance_test_table_8 [id=ea84ae8db01148a6896438374a278971]) successfully deleted
I20240501 08:12:06.564615 31031 ts_tablet_manager.cc:1888] T 3b24297ce5da4790877c5f5672a73d8c P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:06.564822 31032 tablet_replica.cc:330] T 3b24297ce5da4790877c5f5672a73d8c P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:06.564903 31031 log.cc:1196] T 3b24297ce5da4790877c5f5672a73d8c P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/3b24297ce5da4790877c5f5672a73d8c
I20240501 08:12:06.565095 31032 raft_consensus.cc:2229] T 3b24297ce5da4790877c5f5672a73d8c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:06.565358 31032 raft_consensus.cc:2258] T 3b24297ce5da4790877c5f5672a73d8c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:06.565409 31031 ts_tablet_manager.cc:1909] T 3b24297ce5da4790877c5f5672a73d8c P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting consensus metadata
I20240501 08:12:06.566167 31032 ts_tablet_manager.cc:1875] T 3b24297ce5da4790877c5f5672a73d8c P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:06.566612 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet 3b24297ce5da4790877c5f5672a73d8c (table rebalance_test_table_8 [id=ea84ae8db01148a6896438374a278971]) successfully deleted
I20240501 08:12:06.567127 31158 ts_tablet_manager.cc:1888] T 44f9bd2dc83a4533afc4cff8e93164e3 P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:06.567272 31158 log.cc:1196] T 44f9bd2dc83a4533afc4cff8e93164e3 P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/44f9bd2dc83a4533afc4cff8e93164e3
I20240501 08:12:06.567669 31223 ts_tablet_manager.cc:1888] T 44f9bd2dc83a4533afc4cff8e93164e3 P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:06.567713 31158 ts_tablet_manager.cc:1909] T 44f9bd2dc83a4533afc4cff8e93164e3 P aad81047009a4ffcbc450b60030583b5: Deleting consensus metadata
I20240501 08:12:06.567840 31223 log.cc:1196] T 44f9bd2dc83a4533afc4cff8e93164e3 P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/44f9bd2dc83a4533afc4cff8e93164e3
I20240501 08:12:06.568323 31223 ts_tablet_manager.cc:1909] T 44f9bd2dc83a4533afc4cff8e93164e3 P f1525cbc82a24d13b65f136695c903af: Deleting consensus metadata
I20240501 08:12:06.568771 31158 tablet_replica.cc:330] T dfb81aba648d4099a949064b7f4442be P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:06.568843 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet 44f9bd2dc83a4533afc4cff8e93164e3 (table rebalance_test_table_8 [id=ea84ae8db01148a6896438374a278971]) successfully deleted
I20240501 08:12:06.569198 31158 raft_consensus.cc:2229] T dfb81aba648d4099a949064b7f4442be P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:06.569522 31158 raft_consensus.cc:2258] T dfb81aba648d4099a949064b7f4442be P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:06.569511 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet 44f9bd2dc83a4533afc4cff8e93164e3 (table rebalance_test_table_8 [id=ea84ae8db01148a6896438374a278971]) successfully deleted
I20240501 08:12:06.570178 31223 tablet_replica.cc:330] T 3b24297ce5da4790877c5f5672a73d8c P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:06.570302 31158 ts_tablet_manager.cc:1875] T dfb81aba648d4099a949064b7f4442be P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:06.570472 31223 raft_consensus.cc:2229] T 3b24297ce5da4790877c5f5672a73d8c P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:06.570744 31223 raft_consensus.cc:2258] T 3b24297ce5da4790877c5f5672a73d8c P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:06.571652 31223 ts_tablet_manager.cc:1875] T 3b24297ce5da4790877c5f5672a73d8c P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:06.573737 31032 ts_tablet_manager.cc:1888] T 3b24297ce5da4790877c5f5672a73d8c P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:06.574035 31032 log.cc:1196] T 3b24297ce5da4790877c5f5672a73d8c P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/3b24297ce5da4790877c5f5672a73d8c
I20240501 08:12:06.574509 31032 ts_tablet_manager.cc:1909] T 3b24297ce5da4790877c5f5672a73d8c P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting consensus metadata
I20240501 08:12:06.575604 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet 3b24297ce5da4790877c5f5672a73d8c (table rebalance_test_table_8 [id=ea84ae8db01148a6896438374a278971]) successfully deleted
I20240501 08:12:06.577678 31158 ts_tablet_manager.cc:1888] T dfb81aba648d4099a949064b7f4442be P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:06.577872 31158 log.cc:1196] T dfb81aba648d4099a949064b7f4442be P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/dfb81aba648d4099a949064b7f4442be
I20240501 08:12:06.578433 31158 ts_tablet_manager.cc:1909] T dfb81aba648d4099a949064b7f4442be P aad81047009a4ffcbc450b60030583b5: Deleting consensus metadata
I20240501 08:12:06.579936 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet dfb81aba648d4099a949064b7f4442be (table rebalance_test_table_8 [id=ea84ae8db01148a6896438374a278971]) successfully deleted
I20240501 08:12:06.580298 31223 ts_tablet_manager.cc:1888] T 3b24297ce5da4790877c5f5672a73d8c P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:06.580446 31223 log.cc:1196] T 3b24297ce5da4790877c5f5672a73d8c P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/3b24297ce5da4790877c5f5672a73d8c
I20240501 08:12:06.580870 31223 ts_tablet_manager.cc:1909] T 3b24297ce5da4790877c5f5672a73d8c P f1525cbc82a24d13b65f136695c903af: Deleting consensus metadata
I20240501 08:12:06.581775 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet 3b24297ce5da4790877c5f5672a73d8c (table rebalance_test_table_8 [id=ea84ae8db01148a6896438374a278971]) successfully deleted
I20240501 08:12:06.581732 27613 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:06.589071 31241 tablet.cc:1722] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:06.589720 31242 tablet.cc:1722] T 95732c57c31a4f899dab7f60c1c89e48 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:06.589727 31243 tablet.cc:1722] T 95732c57c31a4f899dab7f60c1c89e48 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:06.598184 27613 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table_new_name [id=c3119cae82434187b5457d0f9c9309e8] alter complete (version 3)
I20240501 08:12:06.656826 27617 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_extra_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:06.665068 30652 tablet_service.cc:1467] Processing CreateTablet for tablet 00a706ff4ea7456aacd2586791704e35 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=0d602f5b725a462185dafa660a8e4c1a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:06.665547 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00a706ff4ea7456aacd2586791704e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:06.665511 30517 tablet_service.cc:1467] Processing CreateTablet for tablet 00a706ff4ea7456aacd2586791704e35 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=0d602f5b725a462185dafa660a8e4c1a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:06.665511 30515 tablet_service.cc:1467] Processing CreateTablet for tablet 9df818d6cb4946989f8a4d2e2ce9b8bb (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=0d602f5b725a462185dafa660a8e4c1a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:06.666101 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00a706ff4ea7456aacd2586791704e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:06.666411 28000 tablet_service.cc:1467] Processing CreateTablet for tablet 00a706ff4ea7456aacd2586791704e35 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=0d602f5b725a462185dafa660a8e4c1a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:06.666458 28002 tablet_service.cc:1467] Processing CreateTablet for tablet 9df818d6cb4946989f8a4d2e2ce9b8bb (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=0d602f5b725a462185dafa660a8e4c1a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:06.666801 28000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00a706ff4ea7456aacd2586791704e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:06.667750 30783 tablet_service.cc:1467] Processing CreateTablet for tablet 9df818d6cb4946989f8a4d2e2ce9b8bb (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=0d602f5b725a462185dafa660a8e4c1a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:06.668134 30783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9df818d6cb4946989f8a4d2e2ce9b8bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:06.670027 30515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9df818d6cb4946989f8a4d2e2ce9b8bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:06.670085 28002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9df818d6cb4946989f8a4d2e2ce9b8bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:06.676993 31252 tablet_bootstrap.cc:492] T 00a706ff4ea7456aacd2586791704e35 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:06.679412 31252 tablet_bootstrap.cc:654] T 00a706ff4ea7456aacd2586791704e35 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:06.681397 30517 tablet_service.cc:1467] Processing CreateTablet for tablet 83b422d7436a41e4aa6398963fb40f1e (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=0d602f5b725a462185dafa660a8e4c1a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:06.682216 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83b422d7436a41e4aa6398963fb40f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:06.689471 30916 tablet_service.cc:1467] Processing CreateTablet for tablet 83b422d7436a41e4aa6398963fb40f1e (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=0d602f5b725a462185dafa660a8e4c1a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:06.690121 30916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83b422d7436a41e4aa6398963fb40f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:06.690414 31253 tablet_bootstrap.cc:492] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:06.691694 27870 tablet_service.cc:1467] Processing CreateTablet for tablet 83b422d7436a41e4aa6398963fb40f1e (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=0d602f5b725a462185dafa660a8e4c1a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:06.692152 27870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83b422d7436a41e4aa6398963fb40f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:06.694360 31253 tablet_bootstrap.cc:654] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:06.701109 27620 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:06.701448 27620 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:06.702579 31253 tablet_bootstrap.cc:492] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:06.702823 31253 ts_tablet_manager.cc:1367] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20240501 08:12:06.703681 31253 raft_consensus.cc:348] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:06.704035 31253 raft_consensus.cc:374] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:06.704196 31253 raft_consensus.cc:729] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:06.704545 31253 consensus_queue.cc:260] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:06.705361 31259 tablet_bootstrap.cc:492] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:06.710305 31259 tablet_bootstrap.cc:654] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:06.710523 31253 ts_tablet_manager.cc:1398] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240501 08:12:06.711575 27620 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 95732c57c31a4f899dab7f60c1c89e48
I20240501 08:12:06.712373 31257 tablet_bootstrap.cc:492] T 83b422d7436a41e4aa6398963fb40f1e P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:06.713373 30652 tablet_service.cc:1514] Processing DeleteTablet for tablet 95732c57c31a4f899dab7f60c1c89e48 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:06 UTC) from {username='slave'} at 127.0.0.1:57156
I20240501 08:12:06.714066 28132 tablet_service.cc:1514] Processing DeleteTablet for tablet 95732c57c31a4f899dab7f60c1c89e48 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:06 UTC) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:06.714351 31031 tablet_replica.cc:330] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:06.714562 31031 raft_consensus.cc:2229] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:06.714954 31031 raft_consensus.cc:2258] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:06.715878 31257 tablet_bootstrap.cc:654] T 83b422d7436a41e4aa6398963fb40f1e P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:06.715960 31031 ts_tablet_manager.cc:1875] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:06.718281 31252 tablet_bootstrap.cc:492] T 00a706ff4ea7456aacd2586791704e35 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:06.718608 31252 ts_tablet_manager.cc:1367] T 00a706ff4ea7456aacd2586791704e35 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.042s user 0.018s sys 0.013s
I20240501 08:12:06.719524 31252 raft_consensus.cc:348] T 00a706ff4ea7456aacd2586791704e35 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:06.719877 31252 raft_consensus.cc:374] T 00a706ff4ea7456aacd2586791704e35 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:06.720124 31252 raft_consensus.cc:729] T 00a706ff4ea7456aacd2586791704e35 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:06.720816 31252 consensus_queue.cc:260] T 00a706ff4ea7456aacd2586791704e35 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:06.725047 31252 ts_tablet_manager.cc:1398] T 00a706ff4ea7456aacd2586791704e35 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240501 08:12:06.726855 31031 ts_tablet_manager.cc:1888] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:06.728380 31252 tablet_bootstrap.cc:492] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:06.728432 30783 tablet_service.cc:1514] Processing DeleteTablet for tablet 95732c57c31a4f899dab7f60c1c89e48 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:06 UTC) from {username='slave'} at 127.0.0.1:38952
I20240501 08:12:06.729151 31031 log.cc:1196] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/95732c57c31a4f899dab7f60c1c89e48
I20240501 08:12:06.730809 31031 ts_tablet_manager.cc:1909] T 95732c57c31a4f899dab7f60c1c89e48 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting consensus metadata
I20240501 08:12:06.732105 31259 tablet_bootstrap.cc:492] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:06.732550 31259 ts_tablet_manager.cc:1367] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.003s
I20240501 08:12:06.733551 31258 tablet_bootstrap.cc:492] T 9df818d6cb4946989f8a4d2e2ce9b8bb P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:06.734108 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet 95732c57c31a4f899dab7f60c1c89e48 (table rebalancer_dynamic_table_new_name [id=c3119cae82434187b5457d0f9c9309e8]) successfully deleted
I20240501 08:12:06.736096 31258 tablet_bootstrap.cc:654] T 9df818d6cb4946989f8a4d2e2ce9b8bb P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:06.738375 31260 tablet_replica.cc:330] T 95732c57c31a4f899dab7f60c1c89e48 P 9a022e1a501f4d44a0468054d6c3de47: stopping tablet replica
I20240501 08:12:06.738379 31259 raft_consensus.cc:348] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:06.738806 31259 raft_consensus.cc:374] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:06.738973 31259 raft_consensus.cc:729] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:06.738703 31260 raft_consensus.cc:2229] T 95732c57c31a4f899dab7f60c1c89e48 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:06.739431 31260 raft_consensus.cc:2258] T 95732c57c31a4f899dab7f60c1c89e48 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:06.740453 31252 tablet_bootstrap.cc:654] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:06.740535 31260 ts_tablet_manager.cc:1875] T 95732c57c31a4f899dab7f60c1c89e48 P 9a022e1a501f4d44a0468054d6c3de47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:06.740933 31259 consensus_queue.cc:260] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:06.741204 31263 tablet_bootstrap.cc:492] T 83b422d7436a41e4aa6398963fb40f1e P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:06.742889 31259 ts_tablet_manager.cc:1398] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240501 08:12:06.745010 31263 tablet_bootstrap.cc:654] T 83b422d7436a41e4aa6398963fb40f1e P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:06.750026 31264 tablet_replica.cc:330] T 95732c57c31a4f899dab7f60c1c89e48 P 64998323a1f4482d8704bf20594d13a5: stopping tablet replica
I20240501 08:12:06.750721 31264 raft_consensus.cc:2229] T 95732c57c31a4f899dab7f60c1c89e48 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:06.751627 31260 ts_tablet_manager.cc:1888] T 95732c57c31a4f899dab7f60c1c89e48 P 9a022e1a501f4d44a0468054d6c3de47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:06.752223 31252 tablet_bootstrap.cc:492] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:06.752843 31252 ts_tablet_manager.cc:1367] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.002s
I20240501 08:12:06.753355 31258 tablet_bootstrap.cc:492] T 9df818d6cb4946989f8a4d2e2ce9b8bb P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:12:06.753568 31258 ts_tablet_manager.cc:1367] T 9df818d6cb4946989f8a4d2e2ce9b8bb P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.000s
I20240501 08:12:06.753863 31252 raft_consensus.cc:348] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:06.754194 31252 raft_consensus.cc:374] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:06.754310 31252 raft_consensus.cc:729] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:06.754424 31258 raft_consensus.cc:348] T 9df818d6cb4946989f8a4d2e2ce9b8bb P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:06.754616 31258 raft_consensus.cc:374] T 9df818d6cb4946989f8a4d2e2ce9b8bb P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:06.754710 31258 raft_consensus.cc:729] T 9df818d6cb4946989f8a4d2e2ce9b8bb P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:12:06.754755 31252 consensus_queue.cc:260] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:06.754956 31260 log.cc:1196] T 95732c57c31a4f899dab7f60c1c89e48 P 9a022e1a501f4d44a0468054d6c3de47: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/wals/95732c57c31a4f899dab7f60c1c89e48
I20240501 08:12:06.755276 31258 consensus_queue.cc:260] T 9df818d6cb4946989f8a4d2e2ce9b8bb P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:06.755951 31258 ts_tablet_manager.cc:1398] T 9df818d6cb4946989f8a4d2e2ce9b8bb P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:06.756785 31257 tablet_bootstrap.cc:492] T 83b422d7436a41e4aa6398963fb40f1e P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:06.757123 31258 tablet_bootstrap.cc:492] T 00a706ff4ea7456aacd2586791704e35 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:06.757207 31260 ts_tablet_manager.cc:1909] T 95732c57c31a4f899dab7f60c1c89e48 P 9a022e1a501f4d44a0468054d6c3de47: Deleting consensus metadata
I20240501 08:12:06.758013 31252 ts_tablet_manager.cc:1398] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:12:06.758648 27604 catalog_manager.cc:4830] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): tablet 95732c57c31a4f899dab7f60c1c89e48 (table rebalancer_dynamic_table_new_name [id=c3119cae82434187b5457d0f9c9309e8]) successfully deleted
I20240501 08:12:06.759125 31257 ts_tablet_manager.cc:1367] T 83b422d7436a41e4aa6398963fb40f1e P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.047s user 0.014s sys 0.018s
I20240501 08:12:06.759991 31257 raft_consensus.cc:348] T 83b422d7436a41e4aa6398963fb40f1e P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:06.760171 31264 raft_consensus.cc:2258] T 95732c57c31a4f899dab7f60c1c89e48 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:06.760548 31258 tablet_bootstrap.cc:654] T 00a706ff4ea7456aacd2586791704e35 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:06.760993 31252 tablet_bootstrap.cc:492] T 83b422d7436a41e4aa6398963fb40f1e P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:06.761610 31257 raft_consensus.cc:374] T 83b422d7436a41e4aa6398963fb40f1e P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:06.761765 31257 raft_consensus.cc:729] T 83b422d7436a41e4aa6398963fb40f1e P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:06.762099 31257 consensus_queue.cc:260] T 83b422d7436a41e4aa6398963fb40f1e P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:06.762308 31264 ts_tablet_manager.cc:1875] T 95732c57c31a4f899dab7f60c1c89e48 P 64998323a1f4482d8704bf20594d13a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:06.762960 31252 tablet_bootstrap.cc:654] T 83b422d7436a41e4aa6398963fb40f1e P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:06.763448 31257 ts_tablet_manager.cc:1398] T 83b422d7436a41e4aa6398963fb40f1e P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 08:12:06.764783 31263 tablet_bootstrap.cc:492] T 83b422d7436a41e4aa6398963fb40f1e P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:12:06.765024 31263 ts_tablet_manager.cc:1367] T 83b422d7436a41e4aa6398963fb40f1e P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.000s
I20240501 08:12:06.765978 31263 raft_consensus.cc:348] T 83b422d7436a41e4aa6398963fb40f1e P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:06.766263 31263 raft_consensus.cc:374] T 83b422d7436a41e4aa6398963fb40f1e P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:06.766392 31263 raft_consensus.cc:729] T 83b422d7436a41e4aa6398963fb40f1e P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:12:06.766672 31263 consensus_queue.cc:260] T 83b422d7436a41e4aa6398963fb40f1e P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:06.767417 31263 ts_tablet_manager.cc:1398] T 83b422d7436a41e4aa6398963fb40f1e P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:06.771065 31179 raft_consensus.cc:482] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:06.771284 31179 raft_consensus.cc:504] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:06.772243 31179 leader_election.cc:288] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:06.792575 31264 ts_tablet_manager.cc:1888] T 95732c57c31a4f899dab7f60c1c89e48 P 64998323a1f4482d8704bf20594d13a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:06.792923 31264 log.cc:1196] T 95732c57c31a4f899dab7f60c1c89e48 P 64998323a1f4482d8704bf20594d13a5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal/wals/95732c57c31a4f899dab7f60c1c89e48
I20240501 08:12:06.793979 31264 ts_tablet_manager.cc:1909] T 95732c57c31a4f899dab7f60c1c89e48 P 64998323a1f4482d8704bf20594d13a5: Deleting consensus metadata
I20240501 08:12:06.800284 27605 catalog_manager.cc:4830] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): tablet 95732c57c31a4f899dab7f60c1c89e48 (table rebalancer_dynamic_table_new_name [id=c3119cae82434187b5457d0f9c9309e8]) successfully deleted
I20240501 08:12:06.808168 31258 tablet_bootstrap.cc:492] T 00a706ff4ea7456aacd2586791704e35 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:12:06.808781 31258 ts_tablet_manager.cc:1367] T 00a706ff4ea7456aacd2586791704e35 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.052s user 0.010s sys 0.032s
I20240501 08:12:06.810423 31258 raft_consensus.cc:348] T 00a706ff4ea7456aacd2586791704e35 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:06.810819 31252 tablet_bootstrap.cc:492] T 83b422d7436a41e4aa6398963fb40f1e P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:06.810966 31252 ts_tablet_manager.cc:1367] T 83b422d7436a41e4aa6398963fb40f1e P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.050s user 0.004s sys 0.003s
I20240501 08:12:06.811003 31258 raft_consensus.cc:374] T 00a706ff4ea7456aacd2586791704e35 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:06.811142 31258 raft_consensus.cc:729] T 00a706ff4ea7456aacd2586791704e35 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:12:06.811463 31258 consensus_queue.cc:260] T 00a706ff4ea7456aacd2586791704e35 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:06.811865 31252 raft_consensus.cc:348] T 83b422d7436a41e4aa6398963fb40f1e P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:06.812304 31258 ts_tablet_manager.cc:1398] T 00a706ff4ea7456aacd2586791704e35 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:06.812547 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a706ff4ea7456aacd2586791704e35" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:12:06.812747 28022 raft_consensus.cc:2454] T 00a706ff4ea7456aacd2586791704e35 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 0.
I20240501 08:12:06.813231 30607 leader_election.cc:302] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a022e1a501f4d44a0468054d6c3de47, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:12:06.813540 31252 raft_consensus.cc:374] T 83b422d7436a41e4aa6398963fb40f1e P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:06.813560 31179 raft_consensus.cc:2781] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:06.813659 31179 raft_consensus.cc:482] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:06.813764 31179 raft_consensus.cc:3037] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:06.813781 31252 raft_consensus.cc:729] T 83b422d7436a41e4aa6398963fb40f1e P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:06.814198 31252 consensus_queue.cc:260] T 83b422d7436a41e4aa6398963fb40f1e P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:06.816658 31179 raft_consensus.cc:504] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:06.817349 31252 ts_tablet_manager.cc:1398] T 83b422d7436a41e4aa6398963fb40f1e P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240501 08:12:06.817461 31179 leader_election.cc:288] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 election: Requested vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:06.818058 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a706ff4ea7456aacd2586791704e35" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:12:06.818290 28022 raft_consensus.cc:3037] T 00a706ff4ea7456aacd2586791704e35 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:06.821085 28022 raft_consensus.cc:2454] T 00a706ff4ea7456aacd2586791704e35 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 1.
I20240501 08:12:06.817934 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a706ff4ea7456aacd2586791704e35" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:06.821138 27613 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:06.821494 30607 leader_election.cc:302] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a022e1a501f4d44a0468054d6c3de47, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:12:06.821544 30537 raft_consensus.cc:2454] T 00a706ff4ea7456aacd2586791704e35 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 0.
I20240501 08:12:06.819041 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a706ff4ea7456aacd2586791704e35" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:06.821791 30536 raft_consensus.cc:3037] T 00a706ff4ea7456aacd2586791704e35 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:06.821791 31179 raft_consensus.cc:2781] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:06.825592 30536 raft_consensus.cc:2454] T 00a706ff4ea7456aacd2586791704e35 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 1.
I20240501 08:12:06.832221 30783 tablet_service.cc:1467] Processing CreateTablet for tablet bcffa625d53f416cac63b37de07bae28 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fbec419a81994590a2781a5a78b52627]), partition=
I20240501 08:12:06.832604 27870 tablet_service.cc:1467] Processing CreateTablet for tablet bcffa625d53f416cac63b37de07bae28 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fbec419a81994590a2781a5a78b52627]), partition=
I20240501 08:12:06.832665 30783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcffa625d53f416cac63b37de07bae28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:06.832700 30517 tablet_service.cc:1467] Processing CreateTablet for tablet bcffa625d53f416cac63b37de07bae28 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fbec419a81994590a2781a5a78b52627]), partition=
I20240501 08:12:06.832964 27870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcffa625d53f416cac63b37de07bae28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:06.833081 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcffa625d53f416cac63b37de07bae28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:06.838076 31179 raft_consensus.cc:686] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Becoming Leader. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Running, Role: LEADER
I20240501 08:12:06.838418 31179 consensus_queue.cc:237] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:06.839300 31252 tablet_bootstrap.cc:492] T bcffa625d53f416cac63b37de07bae28 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:06.840600 31259 tablet_bootstrap.cc:492] T bcffa625d53f416cac63b37de07bae28 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:06.840624 31263 tablet_bootstrap.cc:492] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:06.841374 31252 tablet_bootstrap.cc:654] T bcffa625d53f416cac63b37de07bae28 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:06.842150 31259 tablet_bootstrap.cc:654] T bcffa625d53f416cac63b37de07bae28 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:06.842257 31263 tablet_bootstrap.cc:654] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:06.847441 31246 raft_consensus.cc:482] T 83b422d7436a41e4aa6398963fb40f1e P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:06.847659 31246 raft_consensus.cc:504] T 83b422d7436a41e4aa6398963fb40f1e P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:06.848918 31246 leader_election.cc:288] T 83b422d7436a41e4aa6398963fb40f1e P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:06.850446 30937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b422d7436a41e4aa6398963fb40f1e" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:06.850741 30937 raft_consensus.cc:2454] T 83b422d7436a41e4aa6398963fb40f1e P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 0.
I20240501 08:12:06.851455 30472 leader_election.cc:302] T 83b422d7436a41e4aa6398963fb40f1e P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:06.855275 31246 raft_consensus.cc:2781] T 83b422d7436a41e4aa6398963fb40f1e P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:06.855465 31246 raft_consensus.cc:482] T 83b422d7436a41e4aa6398963fb40f1e P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:06.855576 31246 raft_consensus.cc:3037] T 83b422d7436a41e4aa6398963fb40f1e P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:06.858704 31252 tablet_bootstrap.cc:492] T bcffa625d53f416cac63b37de07bae28 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:06.859200 31259 tablet_bootstrap.cc:492] T bcffa625d53f416cac63b37de07bae28 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:06.859514 31259 ts_tablet_manager.cc:1367] T bcffa625d53f416cac63b37de07bae28 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.007s
I20240501 08:12:06.860599 31259 raft_consensus.cc:348] T bcffa625d53f416cac63b37de07bae28 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:06.861068 31259 raft_consensus.cc:374] T bcffa625d53f416cac63b37de07bae28 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:06.861224 31259 raft_consensus.cc:729] T bcffa625d53f416cac63b37de07bae28 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:06.858934 31252 ts_tablet_manager.cc:1367] T bcffa625d53f416cac63b37de07bae28 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.011s
I20240501 08:12:06.861744 31259 consensus_queue.cc:260] T bcffa625d53f416cac63b37de07bae28 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:06.862324 31252 raft_consensus.cc:348] T bcffa625d53f416cac63b37de07bae28 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:06.862632 31252 raft_consensus.cc:374] T bcffa625d53f416cac63b37de07bae28 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:06.862730 31252 raft_consensus.cc:729] T bcffa625d53f416cac63b37de07bae28 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:06.863025 31252 consensus_queue.cc:260] T bcffa625d53f416cac63b37de07bae28 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:06.865443 31259 ts_tablet_manager.cc:1398] T bcffa625d53f416cac63b37de07bae28 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:12:06.870867 31263 tablet_bootstrap.cc:492] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:12:06.871050 31263 ts_tablet_manager.cc:1367] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.010s
I20240501 08:12:06.871029 31246 raft_consensus.cc:504] T 83b422d7436a41e4aa6398963fb40f1e P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:06.871810 31263 raft_consensus.cc:348] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:06.871927 31246 leader_election.cc:288] T 83b422d7436a41e4aa6398963fb40f1e P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 election: Requested vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:06.872092 31263 raft_consensus.cc:374] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:06.872185 31263 raft_consensus.cc:729] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:12:06.872478 30937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b422d7436a41e4aa6398963fb40f1e" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:06.872481 31263 consensus_queue.cc:260] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:06.872737 30937 raft_consensus.cc:3037] T 83b422d7436a41e4aa6398963fb40f1e P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:06.873135 31263 ts_tablet_manager.cc:1398] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:06.878384 30937 raft_consensus.cc:2454] T 83b422d7436a41e4aa6398963fb40f1e P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 1.
I20240501 08:12:06.879688 30472 leader_election.cc:302] T 83b422d7436a41e4aa6398963fb40f1e P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:06.880088 31246 raft_consensus.cc:2781] T 83b422d7436a41e4aa6398963fb40f1e P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:06.883721 31246 raft_consensus.cc:686] T 83b422d7436a41e4aa6398963fb40f1e P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Becoming Leader. State: Replica: 31811708c6774037b82df44024b229b6, State: Running, Role: LEADER
I20240501 08:12:06.884258 31246 consensus_queue.cc:237] T 83b422d7436a41e4aa6398963fb40f1e P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:06.888711 27613 catalog_manager.cc:5484] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134). New cstate: current_term: 1 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:06.893347 31169 raft_consensus.cc:482] T 83b422d7436a41e4aa6398963fb40f1e P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:06.893522 31169 raft_consensus.cc:504] T 83b422d7436a41e4aa6398963fb40f1e P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:06.894404 31169 leader_election.cc:288] T 83b422d7436a41e4aa6398963fb40f1e P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:06.897992 31252 ts_tablet_manager.cc:1398] T bcffa625d53f416cac63b37de07bae28 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.037s user 0.006s sys 0.013s
I20240501 08:12:06.913573 27620 catalog_manager.cc:5484] T 83b422d7436a41e4aa6398963fb40f1e P 31811708c6774037b82df44024b229b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31811708c6774037b82df44024b229b6 (127.21.84.133). New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:06.921149 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b422d7436a41e4aa6398963fb40f1e" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:12:06.921449 27889 raft_consensus.cc:2454] T 83b422d7436a41e4aa6398963fb40f1e P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 0.
I20240501 08:12:06.921975 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b422d7436a41e4aa6398963fb40f1e" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:12:06.922214 27889 raft_consensus.cc:3037] T 83b422d7436a41e4aa6398963fb40f1e P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:06.923434 30937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b422d7436a41e4aa6398963fb40f1e" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:06.923738 30937 raft_consensus.cc:2379] T 83b422d7436a41e4aa6398963fb40f1e P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f1525cbc82a24d13b65f136695c903af in current term 1: Already voted for candidate 31811708c6774037b82df44024b229b6 in this term.
I20240501 08:12:06.924671 27889 raft_consensus.cc:2454] T 83b422d7436a41e4aa6398963fb40f1e P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 1.
I20240501 08:12:06.926548 31205 raft_consensus.cc:482] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:06.926745 31205 raft_consensus.cc:504] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:06.927594 31205 leader_election.cc:288] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:06.928061 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df818d6cb4946989f8a4d2e2ce9b8bb" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:06.928329 30536 raft_consensus.cc:2454] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:06.928515 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b422d7436a41e4aa6398963fb40f1e" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:06.929541 30740 leader_election.cc:302] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 64998323a1f4482d8704bf20594d13a5; no voters:
I20240501 08:12:06.930055 27828 leader_election.cc:302] T 83b422d7436a41e4aa6398963fb40f1e P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1525cbc82a24d13b65f136695c903af; no voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11
I20240501 08:12:06.930327 31169 raft_consensus.cc:2726] T 83b422d7436a41e4aa6398963fb40f1e P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:06.930902 31205 raft_consensus.cc:2781] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:06.931037 31205 raft_consensus.cc:482] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:06.931185 31205 raft_consensus.cc:3037] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:06.934834 31205 raft_consensus.cc:504] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:06.935550 31205 leader_election.cc:288] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Requested vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:06.936045 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df818d6cb4946989f8a4d2e2ce9b8bb" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:06.936288 30537 raft_consensus.cc:3037] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:06.939208 30537 raft_consensus.cc:2454] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:06.940352 30740 leader_election.cc:302] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 64998323a1f4482d8704bf20594d13a5; no voters:
I20240501 08:12:06.940716 31205 raft_consensus.cc:2781] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:06.946246 31205 raft_consensus.cc:686] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Becoming Leader. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Running, Role: LEADER
I20240501 08:12:06.946637 31205 consensus_queue.cc:237] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:06.951362 27620 catalog_manager.cc:5484] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64998323a1f4482d8704bf20594d13a5 (127.21.84.135). New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:06.952054 28019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df818d6cb4946989f8a4d2e2ce9b8bb" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:12:06.952019 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df818d6cb4946989f8a4d2e2ce9b8bb" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:12:06.952301 28019 raft_consensus.cc:3037] T 9df818d6cb4946989f8a4d2e2ce9b8bb P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:06.954926 28019 raft_consensus.cc:2454] T 9df818d6cb4946989f8a4d2e2ce9b8bb P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:06.992504 31169 raft_consensus.cc:482] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:06.992691 31169 raft_consensus.cc:504] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:06.993429 31169 leader_election.cc:288] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:06.993842 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcffa625d53f416cac63b37de07bae28" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:06.993959 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcffa625d53f416cac63b37de07bae28" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:06.994076 30803 raft_consensus.cc:2454] T bcffa625d53f416cac63b37de07bae28 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:12:06.994191 30537 raft_consensus.cc:2454] T bcffa625d53f416cac63b37de07bae28 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:12:06.994488 27826 leader_election.cc:302] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:06.994725 31169 raft_consensus.cc:2781] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:06.994835 31169 raft_consensus.cc:482] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:06.994922 31169 raft_consensus.cc:3037] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:06.997308 31169 raft_consensus.cc:504] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:06.997988 31169 leader_election.cc:288] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Requested vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:06.998373 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcffa625d53f416cac63b37de07bae28" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:06.998594 30803 raft_consensus.cc:3037] T bcffa625d53f416cac63b37de07bae28 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:06.998533 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcffa625d53f416cac63b37de07bae28" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:06.998736 30537 raft_consensus.cc:3037] T bcffa625d53f416cac63b37de07bae28 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:07.001531 30537 raft_consensus.cc:2454] T bcffa625d53f416cac63b37de07bae28 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:12:07.001531 30803 raft_consensus.cc:2454] T bcffa625d53f416cac63b37de07bae28 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:12:07.002058 27828 leader_election.cc:302] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:07.002349 31169 raft_consensus.cc:2781] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:07.002532 31169 raft_consensus.cc:686] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Becoming Leader. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Running, Role: LEADER
I20240501 08:12:07.002831 31169 consensus_queue.cc:237] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:07.006248 27620 catalog_manager.cc:5484] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af reported cstate change: term changed from 0 to 1, leader changed from <none> to f1525cbc82a24d13b65f136695c903af (127.21.84.130). New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:07.047266 27620 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240501 08:12:07.047489 27620 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240501 08:12:07.050962 27620 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet f8b10e23797c42c9b3ce5d45a1487f95
I20240501 08:12:07.051545 27620 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 80dac7e5df9248e1bef33b75c3878477
I20240501 08:12:07.051779 27870 tablet_service.cc:1514] Processing DeleteTablet for tablet f8b10e23797c42c9b3ce5d45a1487f95 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:07 UTC) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:07.052117 31223 tablet_replica.cc:330] T f8b10e23797c42c9b3ce5d45a1487f95 P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:07.052320 31223 raft_consensus.cc:2229] T f8b10e23797c42c9b3ce5d45a1487f95 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:07.052393 28000 tablet_service.cc:1514] Processing DeleteTablet for tablet 80dac7e5df9248e1bef33b75c3878477 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:07 UTC) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:07.052493 27620 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet adf805b1ae304ebea98cfb2bf3e4eca1
I20240501 08:12:07.052604 31223 raft_consensus.cc:2258] T f8b10e23797c42c9b3ce5d45a1487f95 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:07.052846 31158 tablet_replica.cc:330] T 80dac7e5df9248e1bef33b75c3878477 P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:07.053083 31158 raft_consensus.cc:2229] T 80dac7e5df9248e1bef33b75c3878477 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:07.053310 31158 raft_consensus.cc:2258] T 80dac7e5df9248e1bef33b75c3878477 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:07.053401 31223 ts_tablet_manager.cc:1875] T f8b10e23797c42c9b3ce5d45a1487f95 P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:07.053874 31158 ts_tablet_manager.cc:1875] T 80dac7e5df9248e1bef33b75c3878477 P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:07.057734 27739 tablet_service.cc:1514] Processing DeleteTablet for tablet f8b10e23797c42c9b3ce5d45a1487f95 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:07 UTC) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:07.058008 31223 ts_tablet_manager.cc:1888] T f8b10e23797c42c9b3ce5d45a1487f95 P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:07.058133 27741 tablet_service.cc:1514] Processing DeleteTablet for tablet 80dac7e5df9248e1bef33b75c3878477 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:07 UTC) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:07.058347 27740 tablet_service.cc:1514] Processing DeleteTablet for tablet adf805b1ae304ebea98cfb2bf3e4eca1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:07 UTC) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:07.058323 28130 tablet_service.cc:1514] Processing DeleteTablet for tablet 80dac7e5df9248e1bef33b75c3878477 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:07 UTC) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:07.058377 31158 ts_tablet_manager.cc:1888] T 80dac7e5df9248e1bef33b75c3878477 P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:07.058526 31158 log.cc:1196] T 80dac7e5df9248e1bef33b75c3878477 P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/80dac7e5df9248e1bef33b75c3878477
I20240501 08:12:07.058559 31032 tablet_replica.cc:330] T f8b10e23797c42c9b3ce5d45a1487f95 P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:07.058581 31031 tablet_replica.cc:330] T 80dac7e5df9248e1bef33b75c3878477 P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:07.058744 31032 raft_consensus.cc:2229] T f8b10e23797c42c9b3ce5d45a1487f95 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:07.058796 31031 raft_consensus.cc:2229] T 80dac7e5df9248e1bef33b75c3878477 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:07.058981 31158 ts_tablet_manager.cc:1909] T 80dac7e5df9248e1bef33b75c3878477 P aad81047009a4ffcbc450b60030583b5: Deleting consensus metadata
I20240501 08:12:07.058993 31032 raft_consensus.cc:2258] T f8b10e23797c42c9b3ce5d45a1487f95 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:07.059120 31031 raft_consensus.cc:2258] T 80dac7e5df9248e1bef33b75c3878477 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:07.059322 28131 tablet_service.cc:1514] Processing DeleteTablet for tablet adf805b1ae304ebea98cfb2bf3e4eca1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:07 UTC) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:07.059546 28132 tablet_service.cc:1514] Processing DeleteTablet for tablet f8b10e23797c42c9b3ce5d45a1487f95 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:07 UTC) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:07.059702 31032 ts_tablet_manager.cc:1875] T f8b10e23797c42c9b3ce5d45a1487f95 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:07.059796 31031 ts_tablet_manager.cc:1875] T 80dac7e5df9248e1bef33b75c3878477 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:07.059950 27870 tablet_service.cc:1514] Processing DeleteTablet for tablet adf805b1ae304ebea98cfb2bf3e4eca1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:07 UTC) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:07.058156 31223 log.cc:1196] T f8b10e23797c42c9b3ce5d45a1487f95 P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/f8b10e23797c42c9b3ce5d45a1487f95
I20240501 08:12:07.060596 31223 ts_tablet_manager.cc:1909] T f8b10e23797c42c9b3ce5d45a1487f95 P f1525cbc82a24d13b65f136695c903af: Deleting consensus metadata
I20240501 08:12:07.060730 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet 80dac7e5df9248e1bef33b75c3878477 (table rebalance_test_table_10 [id=f32998915447419ebb7965fabf78af27]) successfully deleted
I20240501 08:12:07.061867 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet f8b10e23797c42c9b3ce5d45a1487f95 (table rebalance_test_table_10 [id=f32998915447419ebb7965fabf78af27]) successfully deleted
I20240501 08:12:07.062158 31223 tablet_replica.cc:330] T adf805b1ae304ebea98cfb2bf3e4eca1 P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:07.062373 31223 raft_consensus.cc:2229] T adf805b1ae304ebea98cfb2bf3e4eca1 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:07.062573 31223 raft_consensus.cc:2258] T adf805b1ae304ebea98cfb2bf3e4eca1 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:07.063146 31223 ts_tablet_manager.cc:1875] T adf805b1ae304ebea98cfb2bf3e4eca1 P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:07.066256 31032 ts_tablet_manager.cc:1888] T f8b10e23797c42c9b3ce5d45a1487f95 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:07.066318 31031 ts_tablet_manager.cc:1888] T 80dac7e5df9248e1bef33b75c3878477 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:07.066397 31032 log.cc:1196] T f8b10e23797c42c9b3ce5d45a1487f95 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/f8b10e23797c42c9b3ce5d45a1487f95
I20240501 08:12:07.066458 31031 log.cc:1196] T 80dac7e5df9248e1bef33b75c3878477 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/80dac7e5df9248e1bef33b75c3878477
I20240501 08:12:07.066846 31032 ts_tablet_manager.cc:1909] T f8b10e23797c42c9b3ce5d45a1487f95 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting consensus metadata
I20240501 08:12:07.067004 31031 ts_tablet_manager.cc:1909] T 80dac7e5df9248e1bef33b75c3878477 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting consensus metadata
I20240501 08:12:07.067559 31223 ts_tablet_manager.cc:1888] T adf805b1ae304ebea98cfb2bf3e4eca1 P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:07.067688 31223 log.cc:1196] T adf805b1ae304ebea98cfb2bf3e4eca1 P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/adf805b1ae304ebea98cfb2bf3e4eca1
I20240501 08:12:07.067931 31031 tablet_replica.cc:330] T adf805b1ae304ebea98cfb2bf3e4eca1 P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:07.068002 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet f8b10e23797c42c9b3ce5d45a1487f95 (table rebalance_test_table_10 [id=f32998915447419ebb7965fabf78af27]) successfully deleted
I20240501 08:12:07.068118 31223 ts_tablet_manager.cc:1909] T adf805b1ae304ebea98cfb2bf3e4eca1 P f1525cbc82a24d13b65f136695c903af: Deleting consensus metadata
I20240501 08:12:07.068238 31031 raft_consensus.cc:2229] T adf805b1ae304ebea98cfb2bf3e4eca1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:07.068374 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet 80dac7e5df9248e1bef33b75c3878477 (table rebalance_test_table_10 [id=f32998915447419ebb7965fabf78af27]) successfully deleted
I20240501 08:12:07.068606 31031 raft_consensus.cc:2258] T adf805b1ae304ebea98cfb2bf3e4eca1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:07.068681 31032 tablet_replica.cc:330] T 80dac7e5df9248e1bef33b75c3878477 P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:07.068892 31032 raft_consensus.cc:2229] T 80dac7e5df9248e1bef33b75c3878477 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:07.069005 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet adf805b1ae304ebea98cfb2bf3e4eca1 (table rebalance_test_table_10 [id=f32998915447419ebb7965fabf78af27]) successfully deleted
I20240501 08:12:07.069118 31032 raft_consensus.cc:2258] T 80dac7e5df9248e1bef33b75c3878477 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:07.069468 31031 ts_tablet_manager.cc:1875] T adf805b1ae304ebea98cfb2bf3e4eca1 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:07.069780 31032 ts_tablet_manager.cc:1875] T 80dac7e5df9248e1bef33b75c3878477 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:07.075978 31031 ts_tablet_manager.cc:1888] T adf805b1ae304ebea98cfb2bf3e4eca1 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:07.075965 31032 ts_tablet_manager.cc:1888] T 80dac7e5df9248e1bef33b75c3878477 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:07.076117 31031 log.cc:1196] T adf805b1ae304ebea98cfb2bf3e4eca1 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/adf805b1ae304ebea98cfb2bf3e4eca1
I20240501 08:12:07.076133 31032 log.cc:1196] T 80dac7e5df9248e1bef33b75c3878477 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/80dac7e5df9248e1bef33b75c3878477
I20240501 08:12:07.076577 31031 ts_tablet_manager.cc:1909] T adf805b1ae304ebea98cfb2bf3e4eca1 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting consensus metadata
I20240501 08:12:07.076550 31032 ts_tablet_manager.cc:1909] T 80dac7e5df9248e1bef33b75c3878477 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting consensus metadata
I20240501 08:12:07.077549 31031 tablet_replica.cc:330] T f8b10e23797c42c9b3ce5d45a1487f95 P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:07.077574 31032 tablet_replica.cc:330] T adf805b1ae304ebea98cfb2bf3e4eca1 P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:07.077664 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet 80dac7e5df9248e1bef33b75c3878477 (table rebalance_test_table_10 [id=f32998915447419ebb7965fabf78af27]) successfully deleted
I20240501 08:12:07.077751 31032 raft_consensus.cc:2229] T adf805b1ae304ebea98cfb2bf3e4eca1 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:07.077785 31031 raft_consensus.cc:2229] T f8b10e23797c42c9b3ce5d45a1487f95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:07.077963 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet adf805b1ae304ebea98cfb2bf3e4eca1 (table rebalance_test_table_10 [id=f32998915447419ebb7965fabf78af27]) successfully deleted
I20240501 08:12:07.078008 31032 raft_consensus.cc:2258] T adf805b1ae304ebea98cfb2bf3e4eca1 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:07.078140 31031 raft_consensus.cc:2258] T f8b10e23797c42c9b3ce5d45a1487f95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:07.078692 31032 ts_tablet_manager.cc:1875] T adf805b1ae304ebea98cfb2bf3e4eca1 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:07.078711 31031 ts_tablet_manager.cc:1875] T f8b10e23797c42c9b3ce5d45a1487f95 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:07.085184 31031 ts_tablet_manager.cc:1888] T f8b10e23797c42c9b3ce5d45a1487f95 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:07.085196 31032 ts_tablet_manager.cc:1888] T adf805b1ae304ebea98cfb2bf3e4eca1 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:07.085345 31032 log.cc:1196] T adf805b1ae304ebea98cfb2bf3e4eca1 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/adf805b1ae304ebea98cfb2bf3e4eca1
I20240501 08:12:07.085358 31031 log.cc:1196] T f8b10e23797c42c9b3ce5d45a1487f95 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/f8b10e23797c42c9b3ce5d45a1487f95
I20240501 08:12:07.085747 31031 ts_tablet_manager.cc:1909] T f8b10e23797c42c9b3ce5d45a1487f95 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting consensus metadata
I20240501 08:12:07.085762 31032 ts_tablet_manager.cc:1909] T adf805b1ae304ebea98cfb2bf3e4eca1 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting consensus metadata
I20240501 08:12:07.087005 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet f8b10e23797c42c9b3ce5d45a1487f95 (table rebalance_test_table_10 [id=f32998915447419ebb7965fabf78af27]) successfully deleted
I20240501 08:12:07.087086 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet adf805b1ae304ebea98cfb2bf3e4eca1 (table rebalance_test_table_10 [id=f32998915447419ebb7965fabf78af27]) successfully deleted
I20240501 08:12:07.146754 27620 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:07.153026 30803 raft_consensus.cc:1261] T bcffa625d53f416cac63b37de07bae28 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:07.153050 30537 raft_consensus.cc:1261] T bcffa625d53f416cac63b37de07bae28 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:07.153652 31169 consensus_queue.cc:1035] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:07.153962 31222 consensus_queue.cc:1035] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:07.173049 27620 catalog_manager.cc:5576] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503) does not have the latest schema for tablet bcffa625d53f416cac63b37de07bae28 (table rebalancer_dynamic_table [id=fbec419a81994590a2781a5a78b52627]). Expected version 1 got 0
I20240501 08:12:07.175184 27620 catalog_manager.cc:5576] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293) does not have the latest schema for tablet bcffa625d53f416cac63b37de07bae28 (table rebalancer_dynamic_table [id=fbec419a81994590a2781a5a78b52627]). Expected version 1 got 0
I20240501 08:12:07.176642 31298 tablet.cc:1722] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:07.178427 31298 tablet.cc:1620] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af: MemRowSet was empty: no flush needed.
I20240501 08:12:07.179482 31300 tablet.cc:1722] T bcffa625d53f416cac63b37de07bae28 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:07.179934 27620 catalog_manager.cc:5576] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503) does not have the latest schema for tablet bcffa625d53f416cac63b37de07bae28 (table rebalancer_dynamic_table [id=fbec419a81994590a2781a5a78b52627]). Expected version 1 got 0
I20240501 08:12:07.181176 31300 tablet.cc:1620] T bcffa625d53f416cac63b37de07bae28 P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:07.182775 27613 catalog_manager.cc:5576] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293) does not have the latest schema for tablet bcffa625d53f416cac63b37de07bae28 (table rebalancer_dynamic_table [id=fbec419a81994590a2781a5a78b52627]). Expected version 1 got 0
I20240501 08:12:07.178694 31299 tablet.cc:1722] T bcffa625d53f416cac63b37de07bae28 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:07.185588 31299 tablet.cc:1620] T bcffa625d53f416cac63b37de07bae28 P 64998323a1f4482d8704bf20594d13a5: MemRowSet was empty: no flush needed.
I20240501 08:12:07.186460 27613 catalog_manager.cc:5576] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503) does not have the latest schema for tablet bcffa625d53f416cac63b37de07bae28 (table rebalancer_dynamic_table [id=fbec419a81994590a2781a5a78b52627]). Expected version 1 got 0
I20240501 08:12:07.187230 27620 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=fbec419a81994590a2781a5a78b52627] alter complete (version 1)
I20240501 08:12:07.289438 27617 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:07.297190 31298 tablet.cc:1722] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:07.297730 31298 tablet.cc:1620] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af: MemRowSet was empty: no flush needed.
I20240501 08:12:07.297744 31300 tablet.cc:1722] T bcffa625d53f416cac63b37de07bae28 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:07.298228 31300 tablet.cc:1620] T bcffa625d53f416cac63b37de07bae28 P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:07.298220 31299 tablet.cc:1722] T bcffa625d53f416cac63b37de07bae28 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:07.298672 31299 tablet.cc:1620] T bcffa625d53f416cac63b37de07bae28 P 64998323a1f4482d8704bf20594d13a5: MemRowSet was empty: no flush needed.
I20240501 08:12:07.306890 27613 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=fbec419a81994590a2781a5a78b52627] alter complete (version 2)
I20240501 08:12:07.311731 31274 consensus_queue.cc:1035] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:07.317040 31281 consensus_queue.cc:1035] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:07.333215 31246 consensus_queue.cc:1035] T 83b422d7436a41e4aa6398963fb40f1e P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:07.339888 31282 consensus_queue.cc:1035] T 83b422d7436a41e4aa6398963fb40f1e P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:07.409696 27613 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:07.418080 31298 tablet.cc:1722] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:07.419579 31299 tablet.cc:1722] T bcffa625d53f416cac63b37de07bae28 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:07.419764 31300 tablet.cc:1722] T bcffa625d53f416cac63b37de07bae28 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:07.427531 27613 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table_new_name [id=fbec419a81994590a2781a5a78b52627] alter complete (version 3)
I20240501 08:12:07.463872 31205 consensus_queue.cc:1035] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:07.473202 31313 consensus_queue.cc:1035] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:07.508344 27617 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_extra_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:07.516680 30783 tablet_service.cc:1467] Processing CreateTablet for tablet 131eac1e19264e00b76500d2ba42697a (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=2b1f439c36b24b19bec52d25077c330d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:07.516847 30652 tablet_service.cc:1467] Processing CreateTablet for tablet 131eac1e19264e00b76500d2ba42697a (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=2b1f439c36b24b19bec52d25077c330d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:07.517153 30783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 131eac1e19264e00b76500d2ba42697a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:07.517297 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 131eac1e19264e00b76500d2ba42697a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:07.518307 27740 tablet_service.cc:1467] Processing CreateTablet for tablet 3aa09c722530431384e5e7ee9759230c (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=2b1f439c36b24b19bec52d25077c330d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:07.518738 27740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aa09c722530431384e5e7ee9759230c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:07.521780 31316 tablet_bootstrap.cc:492] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:07.521958 28000 tablet_service.cc:1467] Processing CreateTablet for tablet 28a7c90dfdf4459aae99c552cfba5b4b (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=2b1f439c36b24b19bec52d25077c330d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:07.522389 28000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28a7c90dfdf4459aae99c552cfba5b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:07.523483 31316 tablet_bootstrap.cc:654] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:07.525794 30517 tablet_service.cc:1467] Processing CreateTablet for tablet 131eac1e19264e00b76500d2ba42697a (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=2b1f439c36b24b19bec52d25077c330d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:07.526316 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 131eac1e19264e00b76500d2ba42697a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:07.527096 30784 tablet_service.cc:1467] Processing CreateTablet for tablet 28a7c90dfdf4459aae99c552cfba5b4b (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=2b1f439c36b24b19bec52d25077c330d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:07.527634 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28a7c90dfdf4459aae99c552cfba5b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:07.530004 30515 tablet_service.cc:1467] Processing CreateTablet for tablet 28a7c90dfdf4459aae99c552cfba5b4b (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=2b1f439c36b24b19bec52d25077c330d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:07.530491 30515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28a7c90dfdf4459aae99c552cfba5b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:07.531636 27617 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:07.531999 27617 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:07.533021 30782 tablet_service.cc:1467] Processing CreateTablet for tablet 3aa09c722530431384e5e7ee9759230c (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=2b1f439c36b24b19bec52d25077c330d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:07.533568 30782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aa09c722530431384e5e7ee9759230c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:07.534322 30516 tablet_service.cc:1467] Processing CreateTablet for tablet 3aa09c722530431384e5e7ee9759230c (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=2b1f439c36b24b19bec52d25077c330d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:07.534714 30516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aa09c722530431384e5e7ee9759230c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:07.540683 27617 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet bcffa625d53f416cac63b37de07bae28
I20240501 08:12:07.541886 30514 tablet_service.cc:1514] Processing DeleteTablet for tablet bcffa625d53f416cac63b37de07bae28 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:07 UTC) from {username='slave'} at 127.0.0.1:45078
I20240501 08:12:07.542148 30781 tablet_service.cc:1514] Processing DeleteTablet for tablet bcffa625d53f416cac63b37de07bae28 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:07 UTC) from {username='slave'} at 127.0.0.1:38952
I20240501 08:12:07.542994 27870 tablet_service.cc:1514] Processing DeleteTablet for tablet bcffa625d53f416cac63b37de07bae28 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:07 UTC) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:07.543303 31223 tablet_replica.cc:330] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:07.543577 31223 raft_consensus.cc:2229] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:07.543988 31223 raft_consensus.cc:2258] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:07.545562 31223 ts_tablet_manager.cc:1875] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:07.552716 31317 tablet_bootstrap.cc:492] T 131eac1e19264e00b76500d2ba42697a P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:07.554518 27620 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240501 08:12:07.554749 27620 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240501 08:12:07.556373 31317 tablet_bootstrap.cc:654] T 131eac1e19264e00b76500d2ba42697a P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:07.558220 31223 ts_tablet_manager.cc:1888] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:07.558467 31223 log.cc:1196] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/bcffa625d53f416cac63b37de07bae28
I20240501 08:12:07.558810 27620 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet d3b2e8d327844b9ba2f639872c2981e7
I20240501 08:12:07.559203 31223 ts_tablet_manager.cc:1909] T bcffa625d53f416cac63b37de07bae28 P f1525cbc82a24d13b65f136695c903af: Deleting consensus metadata
I20240501 08:12:07.559341 27620 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 9ba9a204ac9a4c5b8c7bd8c776f49db8
I20240501 08:12:07.559732 27620 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet a1a23302b0ad4101b8095fbf647e65ef
I20240501 08:12:07.561018 27741 tablet_service.cc:1514] Processing DeleteTablet for tablet d3b2e8d327844b9ba2f639872c2981e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:07 UTC) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:07.561117 28002 tablet_service.cc:1514] Processing DeleteTablet for tablet d3b2e8d327844b9ba2f639872c2981e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:07 UTC) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:07.561347 27739 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ba9a204ac9a4c5b8c7bd8c776f49db8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:07 UTC) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:07.561409 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet bcffa625d53f416cac63b37de07bae28 (table rebalancer_dynamic_table_new_name [id=fbec419a81994590a2781a5a78b52627]) successfully deleted
I20240501 08:12:07.561497 28001 tablet_service.cc:1514] Processing DeleteTablet for tablet a1a23302b0ad4101b8095fbf647e65ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:07 UTC) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:07.561627 31032 tablet_replica.cc:330] T d3b2e8d327844b9ba2f639872c2981e7 P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:07.561866 31032 raft_consensus.cc:2229] T d3b2e8d327844b9ba2f639872c2981e7 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:07.562040 27870 tablet_service.cc:1514] Processing DeleteTablet for tablet d3b2e8d327844b9ba2f639872c2981e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:07 UTC) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:07.562139 31032 raft_consensus.cc:2258] T d3b2e8d327844b9ba2f639872c2981e7 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:07.562480 28132 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ba9a204ac9a4c5b8c7bd8c776f49db8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:07 UTC) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:07.562780 31031 tablet_replica.cc:330] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:07.562757 28131 tablet_service.cc:1514] Processing DeleteTablet for tablet a1a23302b0ad4101b8095fbf647e65ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:07 UTC) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:07.562919 31032 ts_tablet_manager.cc:1875] T d3b2e8d327844b9ba2f639872c2981e7 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:07.563010 31031 raft_consensus.cc:2229] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:07.563356 31031 raft_consensus.cc:2258] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:07.564028 31031 ts_tablet_manager.cc:1875] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:07.565014 27872 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ba9a204ac9a4c5b8c7bd8c776f49db8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:07 UTC) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:07.565227 27871 tablet_service.cc:1514] Processing DeleteTablet for tablet a1a23302b0ad4101b8095fbf647e65ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:07 UTC) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:07.570894 31031 ts_tablet_manager.cc:1888] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:07.571120 31031 log.cc:1196] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/9ba9a204ac9a4c5b8c7bd8c776f49db8
I20240501 08:12:07.571422 31323 tablet_replica.cc:330] T d3b2e8d327844b9ba2f639872c2981e7 P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:07.571455 31223 tablet_replica.cc:330] T d3b2e8d327844b9ba2f639872c2981e7 P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:07.571612 31031 ts_tablet_manager.cc:1909] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting consensus metadata
I20240501 08:12:07.571718 31223 raft_consensus.cc:2229] T d3b2e8d327844b9ba2f639872c2981e7 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:07.572041 31321 tablet_replica.cc:330] T bcffa625d53f416cac63b37de07bae28 P 64998323a1f4482d8704bf20594d13a5: stopping tablet replica
I20240501 08:12:07.572058 31223 raft_consensus.cc:2258] T d3b2e8d327844b9ba2f639872c2981e7 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:07.572255 31321 raft_consensus.cc:2229] T bcffa625d53f416cac63b37de07bae28 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:07.572484 31321 raft_consensus.cc:2258] T bcffa625d53f416cac63b37de07bae28 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:07.572710 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet 9ba9a204ac9a4c5b8c7bd8c776f49db8 (table rebalance_test_table_12 [id=f662392c9baa4a02b6a9549ded7caf1b]) successfully deleted
I20240501 08:12:07.572854 31323 raft_consensus.cc:2229] T d3b2e8d327844b9ba2f639872c2981e7 P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:07.573182 31223 ts_tablet_manager.cc:1875] T d3b2e8d327844b9ba2f639872c2981e7 P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:07.573360 31323 raft_consensus.cc:2258] T d3b2e8d327844b9ba2f639872c2981e7 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:07.573395 31031 tablet_replica.cc:330] T a1a23302b0ad4101b8095fbf647e65ef P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:07.573585 31031 raft_consensus.cc:2229] T a1a23302b0ad4101b8095fbf647e65ef P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:07.573788 31031 raft_consensus.cc:2258] T a1a23302b0ad4101b8095fbf647e65ef P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:07.574198 31323 ts_tablet_manager.cc:1875] T d3b2e8d327844b9ba2f639872c2981e7 P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:07.574896 31321 ts_tablet_manager.cc:1875] T bcffa625d53f416cac63b37de07bae28 P 64998323a1f4482d8704bf20594d13a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:07.575032 31032 ts_tablet_manager.cc:1888] T d3b2e8d327844b9ba2f639872c2981e7 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:07.575109 31031 ts_tablet_manager.cc:1875] T a1a23302b0ad4101b8095fbf647e65ef P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:07.575250 31032 log.cc:1196] T d3b2e8d327844b9ba2f639872c2981e7 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/d3b2e8d327844b9ba2f639872c2981e7
I20240501 08:12:07.575691 31032 ts_tablet_manager.cc:1909] T d3b2e8d327844b9ba2f639872c2981e7 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting consensus metadata
I20240501 08:12:07.576109 31320 tablet_replica.cc:330] T bcffa625d53f416cac63b37de07bae28 P 31811708c6774037b82df44024b229b6: stopping tablet replica
I20240501 08:12:07.576700 31032 tablet_replica.cc:330] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:07.576807 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet d3b2e8d327844b9ba2f639872c2981e7 (table rebalance_test_table_12 [id=f662392c9baa4a02b6a9549ded7caf1b]) successfully deleted
I20240501 08:12:07.576941 31032 raft_consensus.cc:2229] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:07.577644 31032 raft_consensus.cc:2258] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:07.578966 31320 raft_consensus.cc:2229] T bcffa625d53f416cac63b37de07bae28 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:07.580302 31320 raft_consensus.cc:2258] T bcffa625d53f416cac63b37de07bae28 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:07.585496 31321 ts_tablet_manager.cc:1888] T bcffa625d53f416cac63b37de07bae28 P 64998323a1f4482d8704bf20594d13a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:07.589653 31032 ts_tablet_manager.cc:1875] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:07.590193 31323 ts_tablet_manager.cc:1888] T d3b2e8d327844b9ba2f639872c2981e7 P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:07.592686 31323 log.cc:1196] T d3b2e8d327844b9ba2f639872c2981e7 P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/d3b2e8d327844b9ba2f639872c2981e7
I20240501 08:12:07.592742 31031 ts_tablet_manager.cc:1888] T a1a23302b0ad4101b8095fbf647e65ef P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:07.590313 31316 tablet_bootstrap.cc:492] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:07.593289 31321 log.cc:1196] T bcffa625d53f416cac63b37de07bae28 P 64998323a1f4482d8704bf20594d13a5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal/wals/bcffa625d53f416cac63b37de07bae28
I20240501 08:12:07.593374 31316 ts_tablet_manager.cc:1367] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.072s user 0.013s sys 0.009s
I20240501 08:12:07.594115 31321 ts_tablet_manager.cc:1909] T bcffa625d53f416cac63b37de07bae28 P 64998323a1f4482d8704bf20594d13a5: Deleting consensus metadata
I20240501 08:12:07.594736 31031 log.cc:1196] T a1a23302b0ad4101b8095fbf647e65ef P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/a1a23302b0ad4101b8095fbf647e65ef
I20240501 08:12:07.594959 31223 ts_tablet_manager.cc:1888] T d3b2e8d327844b9ba2f639872c2981e7 P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:07.595082 31323 ts_tablet_manager.cc:1909] T d3b2e8d327844b9ba2f639872c2981e7 P aad81047009a4ffcbc450b60030583b5: Deleting consensus metadata
I20240501 08:12:07.595171 31223 log.cc:1196] T d3b2e8d327844b9ba2f639872c2981e7 P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/d3b2e8d327844b9ba2f639872c2981e7
I20240501 08:12:07.595247 31031 ts_tablet_manager.cc:1909] T a1a23302b0ad4101b8095fbf647e65ef P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting consensus metadata
I20240501 08:12:07.594488 31316 raft_consensus.cc:348] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:07.595723 31316 raft_consensus.cc:374] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:07.595875 31316 raft_consensus.cc:729] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:07.596220 31316 consensus_queue.cc:260] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:07.596638 27605 catalog_manager.cc:4830] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): tablet bcffa625d53f416cac63b37de07bae28 (table rebalancer_dynamic_table_new_name [id=fbec419a81994590a2781a5a78b52627]) successfully deleted
I20240501 08:12:07.596645 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet d3b2e8d327844b9ba2f639872c2981e7 (table rebalance_test_table_12 [id=f662392c9baa4a02b6a9549ded7caf1b]) successfully deleted
I20240501 08:12:07.597425 31323 tablet_replica.cc:330] T a1a23302b0ad4101b8095fbf647e65ef P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:07.597571 31317 tablet_bootstrap.cc:492] T 131eac1e19264e00b76500d2ba42697a P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:07.597708 31317 ts_tablet_manager.cc:1367] T 131eac1e19264e00b76500d2ba42697a P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.002s
I20240501 08:12:07.598011 31323 raft_consensus.cc:2229] T a1a23302b0ad4101b8095fbf647e65ef P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:07.598320 31316 ts_tablet_manager.cc:1398] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:12:07.598553 31323 raft_consensus.cc:2258] T a1a23302b0ad4101b8095fbf647e65ef P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:07.598667 31316 tablet_bootstrap.cc:492] T 3aa09c722530431384e5e7ee9759230c P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:07.598564 31317 raft_consensus.cc:348] T 131eac1e19264e00b76500d2ba42697a P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:07.598759 31317 raft_consensus.cc:374] T 131eac1e19264e00b76500d2ba42697a P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:07.598917 31317 raft_consensus.cc:729] T 131eac1e19264e00b76500d2ba42697a P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:07.599231 31317 consensus_queue.cc:260] T 131eac1e19264e00b76500d2ba42697a P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:07.600980 31323 ts_tablet_manager.cc:1875] T a1a23302b0ad4101b8095fbf647e65ef P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:07.601305 31324 tablet_bootstrap.cc:492] T 131eac1e19264e00b76500d2ba42697a P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:07.602519 31316 tablet_bootstrap.cc:654] T 3aa09c722530431384e5e7ee9759230c P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:07.603583 31223 ts_tablet_manager.cc:1909] T d3b2e8d327844b9ba2f639872c2981e7 P f1525cbc82a24d13b65f136695c903af: Deleting consensus metadata
I20240501 08:12:07.603603 31324 tablet_bootstrap.cc:654] T 131eac1e19264e00b76500d2ba42697a P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:07.604794 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet d3b2e8d327844b9ba2f639872c2981e7 (table rebalance_test_table_12 [id=f662392c9baa4a02b6a9549ded7caf1b]) successfully deleted
I20240501 08:12:07.604866 31223 tablet_replica.cc:330] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:07.605095 31223 raft_consensus.cc:2229] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:07.605206 31317 ts_tablet_manager.cc:1398] T 131eac1e19264e00b76500d2ba42697a P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 08:12:07.605415 31223 raft_consensus.cc:2258] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:07.605826 31322 tablet_bootstrap.cc:492] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:07.606140 31223 ts_tablet_manager.cc:1875] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:07.607726 31322 tablet_bootstrap.cc:654] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:07.609241 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet a1a23302b0ad4101b8095fbf647e65ef (table rebalance_test_table_12 [id=f662392c9baa4a02b6a9549ded7caf1b]) successfully deleted
I20240501 08:12:07.610522 31323 ts_tablet_manager.cc:1888] T a1a23302b0ad4101b8095fbf647e65ef P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:07.610776 31323 log.cc:1196] T a1a23302b0ad4101b8095fbf647e65ef P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/a1a23302b0ad4101b8095fbf647e65ef
I20240501 08:12:07.611308 31323 ts_tablet_manager.cc:1909] T a1a23302b0ad4101b8095fbf647e65ef P aad81047009a4ffcbc450b60030583b5: Deleting consensus metadata
I20240501 08:12:07.611646 31319 tablet_bootstrap.cc:492] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:07.612490 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet a1a23302b0ad4101b8095fbf647e65ef (table rebalance_test_table_12 [id=f662392c9baa4a02b6a9549ded7caf1b]) successfully deleted
I20240501 08:12:07.613157 31322 tablet_bootstrap.cc:492] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:12:07.613320 31322 ts_tablet_manager.cc:1367] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240501 08:12:07.613332 31319 tablet_bootstrap.cc:654] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:07.614122 31322 raft_consensus.cc:348] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:07.614346 31322 raft_consensus.cc:374] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:07.614459 31322 raft_consensus.cc:729] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:12:07.614488 31320 ts_tablet_manager.cc:1875] T bcffa625d53f416cac63b37de07bae28 P 31811708c6774037b82df44024b229b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:07.614666 31322 consensus_queue.cc:260] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:07.615173 31322 ts_tablet_manager.cc:1398] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:07.618961 31316 tablet_bootstrap.cc:492] T 3aa09c722530431384e5e7ee9759230c P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:07.619225 31316 ts_tablet_manager.cc:1367] T 3aa09c722530431384e5e7ee9759230c P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20240501 08:12:07.620893 31324 tablet_bootstrap.cc:492] T 131eac1e19264e00b76500d2ba42697a P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:07.620972 31320 ts_tablet_manager.cc:1888] T bcffa625d53f416cac63b37de07bae28 P 31811708c6774037b82df44024b229b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:07.621168 31320 log.cc:1196] T bcffa625d53f416cac63b37de07bae28 P 31811708c6774037b82df44024b229b6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/wals/bcffa625d53f416cac63b37de07bae28
I20240501 08:12:07.621145 31316 raft_consensus.cc:348] T 3aa09c722530431384e5e7ee9759230c P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:07.621448 31316 raft_consensus.cc:374] T 3aa09c722530431384e5e7ee9759230c P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:07.621544 31316 raft_consensus.cc:729] T 3aa09c722530431384e5e7ee9759230c P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:07.621986 31320 ts_tablet_manager.cc:1909] T bcffa625d53f416cac63b37de07bae28 P 31811708c6774037b82df44024b229b6: Deleting consensus metadata
I20240501 08:12:07.621865 31316 consensus_queue.cc:260] T 3aa09c722530431384e5e7ee9759230c P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:07.623360 27607 catalog_manager.cc:4830] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): tablet bcffa625d53f416cac63b37de07bae28 (table rebalancer_dynamic_table_new_name [id=fbec419a81994590a2781a5a78b52627]) successfully deleted
I20240501 08:12:07.624078 31316 ts_tablet_manager.cc:1398] T 3aa09c722530431384e5e7ee9759230c P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:12:07.624230 31032 ts_tablet_manager.cc:1888] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:07.624368 31032 log.cc:1196] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/9ba9a204ac9a4c5b8c7bd8c776f49db8
I20240501 08:12:07.621059 31324 ts_tablet_manager.cc:1367] T 131eac1e19264e00b76500d2ba42697a P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20240501 08:12:07.624671 31316 tablet_bootstrap.cc:492] T 28a7c90dfdf4459aae99c552cfba5b4b P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:07.624862 31032 ts_tablet_manager.cc:1909] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting consensus metadata
I20240501 08:12:07.625487 31324 raft_consensus.cc:348] T 131eac1e19264e00b76500d2ba42697a P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:07.625783 31324 raft_consensus.cc:374] T 131eac1e19264e00b76500d2ba42697a P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:07.625926 31324 raft_consensus.cc:729] T 131eac1e19264e00b76500d2ba42697a P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:07.626164 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet 9ba9a204ac9a4c5b8c7bd8c776f49db8 (table rebalance_test_table_12 [id=f662392c9baa4a02b6a9549ded7caf1b]) successfully deleted
I20240501 08:12:07.626302 31324 consensus_queue.cc:260] T 131eac1e19264e00b76500d2ba42697a P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:07.626948 31324 ts_tablet_manager.cc:1398] T 131eac1e19264e00b76500d2ba42697a P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:07.633774 31324 tablet_bootstrap.cc:492] T 3aa09c722530431384e5e7ee9759230c P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:07.636153 31324 tablet_bootstrap.cc:654] T 3aa09c722530431384e5e7ee9759230c P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:07.639703 31223 ts_tablet_manager.cc:1888] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:07.639887 31223 log.cc:1196] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/9ba9a204ac9a4c5b8c7bd8c776f49db8
I20240501 08:12:07.640447 31223 ts_tablet_manager.cc:1909] T 9ba9a204ac9a4c5b8c7bd8c776f49db8 P f1525cbc82a24d13b65f136695c903af: Deleting consensus metadata
I20240501 08:12:07.641870 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet 9ba9a204ac9a4c5b8c7bd8c776f49db8 (table rebalance_test_table_12 [id=f662392c9baa4a02b6a9549ded7caf1b]) successfully deleted
I20240501 08:12:07.643918 31316 tablet_bootstrap.cc:654] T 28a7c90dfdf4459aae99c552cfba5b4b P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:07.646034 27617 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:07.648015 31205 raft_consensus.cc:482] T 3aa09c722530431384e5e7ee9759230c P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:07.648192 31205 raft_consensus.cc:504] T 3aa09c722530431384e5e7ee9759230c P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:07.649032 31205 leader_election.cc:288] T 3aa09c722530431384e5e7ee9759230c P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:07.649418 31319 tablet_bootstrap.cc:492] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:12:07.649576 31319 ts_tablet_manager.cc:1367] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.038s user 0.020s sys 0.007s
I20240501 08:12:07.650336 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aa09c722530431384e5e7ee9759230c" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:07.650410 31319 raft_consensus.cc:348] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:07.650585 31319 raft_consensus.cc:374] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:07.650667 31319 raft_consensus.cc:729] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:12:07.650919 31319 consensus_queue.cc:260] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:07.651095 31223 tablet_replica.cc:330] T a1a23302b0ad4101b8095fbf647e65ef P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:07.651321 31223 raft_consensus.cc:2229] T a1a23302b0ad4101b8095fbf647e65ef P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:07.651542 31223 raft_consensus.cc:2258] T a1a23302b0ad4101b8095fbf647e65ef P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 08:12:07.651762 30740 leader_election.cc:341] T 3aa09c722530431384e5e7ee9759230c P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:07.652128 31292 raft_consensus.cc:482] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:07.652278 31292 raft_consensus.cc:504] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:07.652626 31223 ts_tablet_manager.cc:1875] T a1a23302b0ad4101b8095fbf647e65ef P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:07.653041 31324 tablet_bootstrap.cc:492] T 3aa09c722530431384e5e7ee9759230c P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:07.653178 31292 leader_election.cc:288] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:07.653235 31324 ts_tablet_manager.cc:1367] T 3aa09c722530431384e5e7ee9759230c P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.002s
I20240501 08:12:07.653999 31324 raft_consensus.cc:348] T 3aa09c722530431384e5e7ee9759230c P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:07.654240 31324 raft_consensus.cc:374] T 3aa09c722530431384e5e7ee9759230c P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:07.654309 31324 raft_consensus.cc:729] T 3aa09c722530431384e5e7ee9759230c P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:07.654588 31324 consensus_queue.cc:260] T 3aa09c722530431384e5e7ee9759230c P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:07.654717 31319 ts_tablet_manager.cc:1398] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:12:07.655170 31324 ts_tablet_manager.cc:1398] T 3aa09c722530431384e5e7ee9759230c P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:07.656306 31316 tablet_bootstrap.cc:492] T 28a7c90dfdf4459aae99c552cfba5b4b P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:07.656497 31316 ts_tablet_manager.cc:1367] T 28a7c90dfdf4459aae99c552cfba5b4b P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.002s
I20240501 08:12:07.657305 31316 raft_consensus.cc:348] T 28a7c90dfdf4459aae99c552cfba5b4b P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:07.657524 31316 raft_consensus.cc:374] T 28a7c90dfdf4459aae99c552cfba5b4b P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:07.657655 31316 raft_consensus.cc:729] T 28a7c90dfdf4459aae99c552cfba5b4b P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:07.657891 31316 consensus_queue.cc:260] T 28a7c90dfdf4459aae99c552cfba5b4b P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:07.658509 31316 ts_tablet_manager.cc:1398] T 28a7c90dfdf4459aae99c552cfba5b4b P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:07.660817 31324 tablet_bootstrap.cc:492] T 28a7c90dfdf4459aae99c552cfba5b4b P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:07.662660 31324 tablet_bootstrap.cc:654] T 28a7c90dfdf4459aae99c552cfba5b4b P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:07.663870 31223 ts_tablet_manager.cc:1888] T a1a23302b0ad4101b8095fbf647e65ef P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:07.665717 30652 tablet_service.cc:1467] Processing CreateTablet for tablet 8aae027f329e43d28ccec6be9d1be772 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5a7febcd91ce4b69992c49762a3b4954]), partition=
I20240501 08:12:07.665736 30517 tablet_service.cc:1467] Processing CreateTablet for tablet 8aae027f329e43d28ccec6be9d1be772 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5a7febcd91ce4b69992c49762a3b4954]), partition=
I20240501 08:12:07.666055 30781 tablet_service.cc:1467] Processing CreateTablet for tablet 8aae027f329e43d28ccec6be9d1be772 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5a7febcd91ce4b69992c49762a3b4954]), partition=
I20240501 08:12:07.666167 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aae027f329e43d28ccec6be9d1be772. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:07.666213 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aae027f329e43d28ccec6be9d1be772. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:07.666457 30781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aae027f329e43d28ccec6be9d1be772. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:07.666823 31223 log.cc:1196] T a1a23302b0ad4101b8095fbf647e65ef P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/a1a23302b0ad4101b8095fbf647e65ef
I20240501 08:12:07.667274 31223 ts_tablet_manager.cc:1909] T a1a23302b0ad4101b8095fbf647e65ef P f1525cbc82a24d13b65f136695c903af: Deleting consensus metadata
I20240501 08:12:07.670099 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet a1a23302b0ad4101b8095fbf647e65ef (table rebalance_test_table_12 [id=f662392c9baa4a02b6a9549ded7caf1b]) successfully deleted
I20240501 08:12:07.671663 31317 tablet_bootstrap.cc:492] T 8aae027f329e43d28ccec6be9d1be772 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:07.672609 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aa09c722530431384e5e7ee9759230c" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:07.672868 30537 raft_consensus.cc:2454] T 3aa09c722530431384e5e7ee9759230c P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:12:07.673286 31317 tablet_bootstrap.cc:654] T 8aae027f329e43d28ccec6be9d1be772 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:07.675096 27697 leader_election.cc:302] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 6331f77dae9d452184e8cd0cbf5f1b43; no voters:
I20240501 08:12:07.675380 31292 raft_consensus.cc:2781] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:07.675504 31292 raft_consensus.cc:482] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:07.675611 31292 raft_consensus.cc:3037] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:07.678911 31316 tablet_bootstrap.cc:492] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:07.678752 31292 raft_consensus.cc:504] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:07.679677 31292 leader_election.cc:288] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Requested vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:07.680003 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aa09c722530431384e5e7ee9759230c" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:07.680343 30537 raft_consensus.cc:3037] T 3aa09c722530431384e5e7ee9759230c P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:07.682502 30537 raft_consensus.cc:2454] T 3aa09c722530431384e5e7ee9759230c P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:12:07.683090 27697 leader_election.cc:302] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 6331f77dae9d452184e8cd0cbf5f1b43; no voters:
I20240501 08:12:07.683480 31292 raft_consensus.cc:2781] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:07.683633 31292 raft_consensus.cc:686] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:12:07.683943 31292 consensus_queue.cc:237] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:07.684419 31316 tablet_bootstrap.cc:654] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:07.687280 31317 tablet_bootstrap.cc:492] T 8aae027f329e43d28ccec6be9d1be772 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:07.687510 31317 ts_tablet_manager.cc:1367] T 8aae027f329e43d28ccec6be9d1be772 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.003s
I20240501 08:12:07.689440 27617 catalog_manager.cc:5484] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:07.690132 31316 tablet_bootstrap.cc:492] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:07.690321 31316 ts_tablet_manager.cc:1367] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.001s
I20240501 08:12:07.691072 31316 raft_consensus.cc:348] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:07.691329 31316 raft_consensus.cc:374] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:07.691442 31316 raft_consensus.cc:729] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:07.691741 31316 consensus_queue.cc:260] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:07.693749 31316 ts_tablet_manager.cc:1398] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:12:07.695312 31317 raft_consensus.cc:348] T 8aae027f329e43d28ccec6be9d1be772 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:07.695588 31317 raft_consensus.cc:374] T 8aae027f329e43d28ccec6be9d1be772 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:07.695703 31317 raft_consensus.cc:729] T 8aae027f329e43d28ccec6be9d1be772 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:07.695904 31324 tablet_bootstrap.cc:492] T 28a7c90dfdf4459aae99c552cfba5b4b P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:07.696012 31317 consensus_queue.cc:260] T 8aae027f329e43d28ccec6be9d1be772 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:07.696133 31324 ts_tablet_manager.cc:1367] T 28a7c90dfdf4459aae99c552cfba5b4b P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.000s
I20240501 08:12:07.696588 31317 ts_tablet_manager.cc:1398] T 8aae027f329e43d28ccec6be9d1be772 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:07.694470 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aa09c722530431384e5e7ee9759230c" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:07.698137 30803 raft_consensus.cc:2454] T 3aa09c722530431384e5e7ee9759230c P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:12:07.694715 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aa09c722530431384e5e7ee9759230c" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:07.698508 30804 raft_consensus.cc:3037] T 3aa09c722530431384e5e7ee9759230c P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:07.701225 30804 raft_consensus.cc:2454] T 3aa09c722530431384e5e7ee9759230c P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:12:07.701737 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aa09c722530431384e5e7ee9759230c" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:12:07.702376 30739 leader_election.cc:302] T 3aa09c722530431384e5e7ee9759230c P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5; no voters: 31811708c6774037b82df44024b229b6, 6331f77dae9d452184e8cd0cbf5f1b43
I20240501 08:12:07.702687 31205 raft_consensus.cc:2726] T 3aa09c722530431384e5e7ee9759230c P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:07.697885 31324 raft_consensus.cc:348] T 28a7c90dfdf4459aae99c552cfba5b4b P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:07.703047 31324 raft_consensus.cc:374] T 28a7c90dfdf4459aae99c552cfba5b4b P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:07.703179 31324 raft_consensus.cc:729] T 28a7c90dfdf4459aae99c552cfba5b4b P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:07.703434 31324 consensus_queue.cc:260] T 28a7c90dfdf4459aae99c552cfba5b4b P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:07.704097 31324 ts_tablet_manager.cc:1398] T 28a7c90dfdf4459aae99c552cfba5b4b P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240501 08:12:07.704594 31324 tablet_bootstrap.cc:492] T 8aae027f329e43d28ccec6be9d1be772 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:07.706262 31324 tablet_bootstrap.cc:654] T 8aae027f329e43d28ccec6be9d1be772 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:07.711222 31324 tablet_bootstrap.cc:492] T 8aae027f329e43d28ccec6be9d1be772 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:07.711367 31324 ts_tablet_manager.cc:1367] T 8aae027f329e43d28ccec6be9d1be772 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20240501 08:12:07.711963 31324 raft_consensus.cc:348] T 8aae027f329e43d28ccec6be9d1be772 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:07.712185 31324 raft_consensus.cc:374] T 8aae027f329e43d28ccec6be9d1be772 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:07.712260 31324 raft_consensus.cc:729] T 8aae027f329e43d28ccec6be9d1be772 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:07.712471 31324 consensus_queue.cc:260] T 8aae027f329e43d28ccec6be9d1be772 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:07.712941 31324 ts_tablet_manager.cc:1398] T 8aae027f329e43d28ccec6be9d1be772 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 08:12:07.715229 31254 raft_consensus.cc:482] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:07.715404 31254 raft_consensus.cc:504] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:07.716444 31254 leader_election.cc:288] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:07.723076 31205 raft_consensus.cc:482] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:07.723292 31205 raft_consensus.cc:504] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:07.723989 31205 leader_election.cc:288] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:07.724421 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "131eac1e19264e00b76500d2ba42697a" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:07.724651 30672 raft_consensus.cc:2454] T 131eac1e19264e00b76500d2ba42697a P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:07.724620 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "131eac1e19264e00b76500d2ba42697a" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:07.724835 30537 raft_consensus.cc:2454] T 131eac1e19264e00b76500d2ba42697a P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:07.725291 30740 leader_election.cc:302] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 64998323a1f4482d8704bf20594d13a5; no voters:
I20240501 08:12:07.725584 31205 raft_consensus.cc:2781] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:07.725674 31205 raft_consensus.cc:482] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:07.725795 31205 raft_consensus.cc:3037] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:07.729379 31205 raft_consensus.cc:504] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:07.730621 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "131eac1e19264e00b76500d2ba42697a" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:07.730852 30672 raft_consensus.cc:3037] T 131eac1e19264e00b76500d2ba42697a P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:07.731175 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "131eac1e19264e00b76500d2ba42697a" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:07.731386 30537 raft_consensus.cc:3037] T 131eac1e19264e00b76500d2ba42697a P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:07.734346 30537 raft_consensus.cc:2454] T 131eac1e19264e00b76500d2ba42697a P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:07.734978 30740 leader_election.cc:302] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 64998323a1f4482d8704bf20594d13a5; no voters:
I20240501 08:12:07.735411 31312 raft_consensus.cc:2781] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:07.735584 31312 raft_consensus.cc:686] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Becoming Leader. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Running, Role: LEADER
I20240501 08:12:07.735970 31312 consensus_queue.cc:237] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:07.735577 31205 leader_election.cc:288] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Requested vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:07.737915 30672 raft_consensus.cc:2454] T 131eac1e19264e00b76500d2ba42697a P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:07.739405 27613 catalog_manager.cc:5484] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64998323a1f4482d8704bf20594d13a5 (127.21.84.135). New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:07.742120 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28a7c90dfdf4459aae99c552cfba5b4b" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:07.742344 30804 raft_consensus.cc:2454] T 28a7c90dfdf4459aae99c552cfba5b4b P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 0.
I20240501 08:12:07.742740 27956 leader_election.cc:302] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:12:07.743105 31254 raft_consensus.cc:2781] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:07.743242 31254 raft_consensus.cc:482] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:07.743326 31254 raft_consensus.cc:3037] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:07.745632 31254 raft_consensus.cc:504] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:07.746371 31254 leader_election.cc:288] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Requested vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:07.746711 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28a7c90dfdf4459aae99c552cfba5b4b" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:07.746946 30804 raft_consensus.cc:3037] T 28a7c90dfdf4459aae99c552cfba5b4b P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:07.747663 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28a7c90dfdf4459aae99c552cfba5b4b" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:07.747788 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28a7c90dfdf4459aae99c552cfba5b4b" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:07.747884 30537 raft_consensus.cc:2454] T 28a7c90dfdf4459aae99c552cfba5b4b P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 0.
I20240501 08:12:07.749246 30804 raft_consensus.cc:2454] T 28a7c90dfdf4459aae99c552cfba5b4b P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 1.
I20240501 08:12:07.749595 27956 leader_election.cc:302] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, aad81047009a4ffcbc450b60030583b5; no voters: 31811708c6774037b82df44024b229b6
I20240501 08:12:07.749954 31254 raft_consensus.cc:2781] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:07.750365 31254 raft_consensus.cc:686] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Becoming Leader. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Running, Role: LEADER
I20240501 08:12:07.750659 31254 consensus_queue.cc:237] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:07.754343 27613 catalog_manager.cc:5484] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to aad81047009a4ffcbc450b60030583b5 (127.21.84.131). New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:07.902283 31205 raft_consensus.cc:482] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:07.902501 31205 raft_consensus.cc:504] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:07.903126 31205 leader_election.cc:288] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:07.903561 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aae027f329e43d28ccec6be9d1be772" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:07.903627 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aae027f329e43d28ccec6be9d1be772" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:07.903784 30672 raft_consensus.cc:2454] T 8aae027f329e43d28ccec6be9d1be772 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:07.903856 30536 raft_consensus.cc:2454] T 8aae027f329e43d28ccec6be9d1be772 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:07.904167 30737 leader_election.cc:302] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:07.904446 31205 raft_consensus.cc:2781] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:07.904536 31205 raft_consensus.cc:482] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:07.904645 31205 raft_consensus.cc:3037] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:07.906607 31205 raft_consensus.cc:504] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:07.907248 31205 leader_election.cc:288] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Requested vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:07.907591 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aae027f329e43d28ccec6be9d1be772" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:07.907686 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aae027f329e43d28ccec6be9d1be772" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:07.907785 30672 raft_consensus.cc:3037] T 8aae027f329e43d28ccec6be9d1be772 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:07.907877 30536 raft_consensus.cc:3037] T 8aae027f329e43d28ccec6be9d1be772 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:07.909996 30672 raft_consensus.cc:2454] T 8aae027f329e43d28ccec6be9d1be772 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:07.910327 30536 raft_consensus.cc:2454] T 8aae027f329e43d28ccec6be9d1be772 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:07.910379 30737 leader_election.cc:302] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:07.910620 31205 raft_consensus.cc:2781] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:07.910784 31205 raft_consensus.cc:686] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Becoming Leader. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Running, Role: LEADER
I20240501 08:12:07.911029 31205 consensus_queue.cc:237] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:07.913736 27613 catalog_manager.cc:5484] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64998323a1f4482d8704bf20594d13a5 (127.21.84.135). New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:08.029078 27613 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:08.035426 30672 raft_consensus.cc:1261] T 8aae027f329e43d28ccec6be9d1be772 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:08.035631 30536 raft_consensus.cc:1261] T 8aae027f329e43d28ccec6be9d1be772 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:08.035950 31312 consensus_queue.cc:1035] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:08.036244 31205 consensus_queue.cc:1035] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:08.041608 27617 catalog_manager.cc:5576] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933) does not have the latest schema for tablet 8aae027f329e43d28ccec6be9d1be772 (table rebalancer_dynamic_table [id=5a7febcd91ce4b69992c49762a3b4954]). Expected version 1 got 0
I20240501 08:12:08.050287 31358 tablet.cc:1722] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:08.050920 31358 tablet.cc:1620] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5: MemRowSet was empty: no flush needed.
I20240501 08:12:08.051702 31359 tablet.cc:1722] T 8aae027f329e43d28ccec6be9d1be772 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:08.052296 31359 tablet.cc:1620] T 8aae027f329e43d28ccec6be9d1be772 P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:08.054654 27613 catalog_manager.cc:5576] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933) does not have the latest schema for tablet 8aae027f329e43d28ccec6be9d1be772 (table rebalancer_dynamic_table [id=5a7febcd91ce4b69992c49762a3b4954]). Expected version 1 got 0
I20240501 08:12:08.056528 27617 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=5a7febcd91ce4b69992c49762a3b4954] alter complete (version 1)
I20240501 08:12:08.063738 27613 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240501 08:12:08.063961 27613 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240501 08:12:08.067600 27613 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 60d21fc9d346438fb4ff01142fd3d255
I20240501 08:12:08.068116 27613 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet eadf85174d8746c691e825949dbbd678
I20240501 08:12:08.068511 27613 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 1b2a23cb9a574422ac3fd3a0d9bcb782
I20240501 08:12:08.068580 27871 tablet_service.cc:1514] Processing DeleteTablet for tablet 60d21fc9d346438fb4ff01142fd3d255 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:08 UTC) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:08.068929 31223 tablet_replica.cc:330] T 60d21fc9d346438fb4ff01142fd3d255 P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:08.069136 31223 raft_consensus.cc:2229] T 60d21fc9d346438fb4ff01142fd3d255 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:08.069422 31223 raft_consensus.cc:2258] T 60d21fc9d346438fb4ff01142fd3d255 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:08.070209 31223 ts_tablet_manager.cc:1875] T 60d21fc9d346438fb4ff01142fd3d255 P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:08.070192 28131 tablet_service.cc:1514] Processing DeleteTablet for tablet 60d21fc9d346438fb4ff01142fd3d255 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:08 UTC) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:08.070431 28132 tablet_service.cc:1514] Processing DeleteTablet for tablet eadf85174d8746c691e825949dbbd678 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:08 UTC) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:08.070529 31031 tablet_replica.cc:330] T 60d21fc9d346438fb4ff01142fd3d255 P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:08.070729 31031 raft_consensus.cc:2229] T 60d21fc9d346438fb4ff01142fd3d255 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:08.071077 31031 raft_consensus.cc:2258] T 60d21fc9d346438fb4ff01142fd3d255 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:08.071763 31031 ts_tablet_manager.cc:1875] T 60d21fc9d346438fb4ff01142fd3d255 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:08.072556 27872 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b2a23cb9a574422ac3fd3a0d9bcb782 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:08 UTC) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:08.073118 27871 tablet_service.cc:1514] Processing DeleteTablet for tablet eadf85174d8746c691e825949dbbd678 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:08 UTC) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:08.073859 28000 tablet_service.cc:1514] Processing DeleteTablet for tablet eadf85174d8746c691e825949dbbd678 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:08 UTC) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:08.074177 27740 tablet_service.cc:1514] Processing DeleteTablet for tablet 60d21fc9d346438fb4ff01142fd3d255 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:08 UTC) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:08.074178 27739 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b2a23cb9a574422ac3fd3a0d9bcb782 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:08 UTC) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:08.074898 31323 tablet_replica.cc:330] T eadf85174d8746c691e825949dbbd678 P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:08.075091 31323 raft_consensus.cc:2229] T eadf85174d8746c691e825949dbbd678 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:08.075327 31323 raft_consensus.cc:2258] T eadf85174d8746c691e825949dbbd678 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:08.076038 31323 ts_tablet_manager.cc:1875] T eadf85174d8746c691e825949dbbd678 P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:08.076354 27613 catalog_manager.cc:5576] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293) does not have the latest schema for tablet 8aae027f329e43d28ccec6be9d1be772 (table rebalancer_dynamic_table [id=5a7febcd91ce4b69992c49762a3b4954]). Expected version 1 got 0
I20240501 08:12:08.078586 28132 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b2a23cb9a574422ac3fd3a0d9bcb782 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:08 UTC) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:08.078773 31223 ts_tablet_manager.cc:1888] T 60d21fc9d346438fb4ff01142fd3d255 P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:08.078907 31223 log.cc:1196] T 60d21fc9d346438fb4ff01142fd3d255 P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/60d21fc9d346438fb4ff01142fd3d255
I20240501 08:12:08.079275 31032 tablet_replica.cc:330] T 60d21fc9d346438fb4ff01142fd3d255 P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:08.079355 31223 ts_tablet_manager.cc:1909] T 60d21fc9d346438fb4ff01142fd3d255 P f1525cbc82a24d13b65f136695c903af: Deleting consensus metadata
I20240501 08:12:08.079512 31032 raft_consensus.cc:2229] T 60d21fc9d346438fb4ff01142fd3d255 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:08.079746 31032 raft_consensus.cc:2258] T 60d21fc9d346438fb4ff01142fd3d255 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:08.080675 31032 ts_tablet_manager.cc:1875] T 60d21fc9d346438fb4ff01142fd3d255 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:08.080886 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet 60d21fc9d346438fb4ff01142fd3d255 (table rebalance_test_table_14 [id=eb27ffc4ce874520a9b72904cae223b2]) successfully deleted
I20240501 08:12:08.081419 31223 tablet_replica.cc:330] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:08.081648 31223 raft_consensus.cc:2229] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:08.081851 31223 raft_consensus.cc:2258] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:08.082592 31223 ts_tablet_manager.cc:1875] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:08.087148 31031 ts_tablet_manager.cc:1888] T 60d21fc9d346438fb4ff01142fd3d255 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:08.087299 31031 log.cc:1196] T 60d21fc9d346438fb4ff01142fd3d255 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/60d21fc9d346438fb4ff01142fd3d255
I20240501 08:12:08.087731 31031 ts_tablet_manager.cc:1909] T 60d21fc9d346438fb4ff01142fd3d255 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting consensus metadata
I20240501 08:12:08.088846 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet 60d21fc9d346438fb4ff01142fd3d255 (table rebalance_test_table_14 [id=eb27ffc4ce874520a9b72904cae223b2]) successfully deleted
I20240501 08:12:08.088877 31292 consensus_queue.cc:1035] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:08.090140 31031 tablet_replica.cc:330] T eadf85174d8746c691e825949dbbd678 P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:08.090375 31031 raft_consensus.cc:2229] T eadf85174d8746c691e825949dbbd678 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:08.090610 31031 raft_consensus.cc:2258] T eadf85174d8746c691e825949dbbd678 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:08.091296 31032 ts_tablet_manager.cc:1888] T 60d21fc9d346438fb4ff01142fd3d255 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:08.091578 31031 ts_tablet_manager.cc:1875] T eadf85174d8746c691e825949dbbd678 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:08.092202 31323 ts_tablet_manager.cc:1888] T eadf85174d8746c691e825949dbbd678 P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:08.092399 31032 log.cc:1196] T 60d21fc9d346438fb4ff01142fd3d255 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/60d21fc9d346438fb4ff01142fd3d255
I20240501 08:12:08.092880 31032 ts_tablet_manager.cc:1909] T 60d21fc9d346438fb4ff01142fd3d255 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting consensus metadata
I20240501 08:12:08.093022 31323 log.cc:1196] T eadf85174d8746c691e825949dbbd678 P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/eadf85174d8746c691e825949dbbd678
I20240501 08:12:08.093479 31323 ts_tablet_manager.cc:1909] T eadf85174d8746c691e825949dbbd678 P aad81047009a4ffcbc450b60030583b5: Deleting consensus metadata
I20240501 08:12:08.094537 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet 60d21fc9d346438fb4ff01142fd3d255 (table rebalance_test_table_14 [id=eb27ffc4ce874520a9b72904cae223b2]) successfully deleted
I20240501 08:12:08.095551 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet eadf85174d8746c691e825949dbbd678 (table rebalance_test_table_14 [id=eb27ffc4ce874520a9b72904cae223b2]) successfully deleted
I20240501 08:12:08.095868 31032 tablet_replica.cc:330] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:08.096103 31032 raft_consensus.cc:2229] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:08.101385 31032 raft_consensus.cc:2258] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:08.102242 31032 ts_tablet_manager.cc:1875] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:08.106143 31223 ts_tablet_manager.cc:1888] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:08.106318 31223 log.cc:1196] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/1b2a23cb9a574422ac3fd3a0d9bcb782
I20240501 08:12:08.108884 31031 ts_tablet_manager.cc:1888] T eadf85174d8746c691e825949dbbd678 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:08.109766 31031 log.cc:1196] T eadf85174d8746c691e825949dbbd678 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/eadf85174d8746c691e825949dbbd678
I20240501 08:12:08.110350 31292 consensus_queue.cc:1035] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:08.110992 31032 ts_tablet_manager.cc:1888] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:08.111157 31032 log.cc:1196] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/1b2a23cb9a574422ac3fd3a0d9bcb782
I20240501 08:12:08.111591 31032 ts_tablet_manager.cc:1909] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting consensus metadata
I20240501 08:12:08.111922 31031 ts_tablet_manager.cc:1909] T eadf85174d8746c691e825949dbbd678 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting consensus metadata
I20240501 08:12:08.112648 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet 1b2a23cb9a574422ac3fd3a0d9bcb782 (table rebalance_test_table_14 [id=eb27ffc4ce874520a9b72904cae223b2]) successfully deleted
I20240501 08:12:08.113090 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet eadf85174d8746c691e825949dbbd678 (table rebalance_test_table_14 [id=eb27ffc4ce874520a9b72904cae223b2]) successfully deleted
I20240501 08:12:08.113559 31223 ts_tablet_manager.cc:1909] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P f1525cbc82a24d13b65f136695c903af: Deleting consensus metadata
I20240501 08:12:08.114571 31223 tablet_replica.cc:330] T eadf85174d8746c691e825949dbbd678 P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:08.114641 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet 1b2a23cb9a574422ac3fd3a0d9bcb782 (table rebalance_test_table_14 [id=eb27ffc4ce874520a9b72904cae223b2]) successfully deleted
I20240501 08:12:08.114876 31223 raft_consensus.cc:2229] T eadf85174d8746c691e825949dbbd678 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:08.114941 31031 tablet_replica.cc:330] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:08.115222 31031 raft_consensus.cc:2229] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:08.115609 31031 raft_consensus.cc:2258] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:08.115736 31223 raft_consensus.cc:2258] T eadf85174d8746c691e825949dbbd678 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:08.116465 31223 ts_tablet_manager.cc:1875] T eadf85174d8746c691e825949dbbd678 P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:08.116575 31360 tablet.cc:1722] T 8aae027f329e43d28ccec6be9d1be772 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:08.116499 31031 ts_tablet_manager.cc:1875] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:08.118499 31360 tablet.cc:1620] T 8aae027f329e43d28ccec6be9d1be772 P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:08.119027 27613 catalog_manager.cc:5576] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293) does not have the latest schema for tablet 8aae027f329e43d28ccec6be9d1be772 (table rebalancer_dynamic_table [id=5a7febcd91ce4b69992c49762a3b4954]). Expected version 1 got 0
I20240501 08:12:08.125919 31223 ts_tablet_manager.cc:1888] T eadf85174d8746c691e825949dbbd678 P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:08.126199 31223 log.cc:1196] T eadf85174d8746c691e825949dbbd678 P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/eadf85174d8746c691e825949dbbd678
I20240501 08:12:08.126732 31223 ts_tablet_manager.cc:1909] T eadf85174d8746c691e825949dbbd678 P f1525cbc82a24d13b65f136695c903af: Deleting consensus metadata
I20240501 08:12:08.126968 31031 ts_tablet_manager.cc:1888] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:08.127139 31031 log.cc:1196] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/1b2a23cb9a574422ac3fd3a0d9bcb782
I20240501 08:12:08.127624 31031 ts_tablet_manager.cc:1909] T 1b2a23cb9a574422ac3fd3a0d9bcb782 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting consensus metadata
I20240501 08:12:08.127955 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet eadf85174d8746c691e825949dbbd678 (table rebalance_test_table_14 [id=eb27ffc4ce874520a9b72904cae223b2]) successfully deleted
I20240501 08:12:08.133715 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet 1b2a23cb9a574422ac3fd3a0d9bcb782 (table rebalance_test_table_14 [id=eb27ffc4ce874520a9b72904cae223b2]) successfully deleted
I20240501 08:12:08.164660 27613 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:08.172330 31358 tablet.cc:1722] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:08.172487 31314 consensus_queue.cc:1035] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:08.172899 31358 tablet.cc:1620] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5: MemRowSet was empty: no flush needed.
I20240501 08:12:08.173600 31360 tablet.cc:1722] T 8aae027f329e43d28ccec6be9d1be772 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:08.174130 31360 tablet.cc:1620] T 8aae027f329e43d28ccec6be9d1be772 P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:08.174490 31359 tablet.cc:1722] T 8aae027f329e43d28ccec6be9d1be772 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:08.174923 31359 tablet.cc:1620] T 8aae027f329e43d28ccec6be9d1be772 P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:08.178107 31254 consensus_queue.cc:1035] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:08.185225 31205 consensus_queue.cc:1035] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:08.189894 27617 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=5a7febcd91ce4b69992c49762a3b4954] alter complete (version 2)
I20240501 08:12:08.196759 30536 raft_consensus.cc:3037] T 28a7c90dfdf4459aae99c552cfba5b4b P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:08.199724 31371 consensus_queue.cc:1035] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:08.269515 27617 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_extra_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:08.277807 30781 tablet_service.cc:1467] Processing CreateTablet for tablet c832fea33e4744868b69e6c29e4aea9f (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=f51464bcf46d406a807b68e0429a75af]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:08.277997 28000 tablet_service.cc:1467] Processing CreateTablet for tablet c832fea33e4744868b69e6c29e4aea9f (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=f51464bcf46d406a807b68e0429a75af]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:08.278348 30781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c832fea33e4744868b69e6c29e4aea9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:08.278235 30916 tablet_service.cc:1467] Processing CreateTablet for tablet c832fea33e4744868b69e6c29e4aea9f (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=f51464bcf46d406a807b68e0429a75af]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:08.278456 28000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c832fea33e4744868b69e6c29e4aea9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:08.278642 30916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c832fea33e4744868b69e6c29e4aea9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:08.281754 27740 tablet_service.cc:1467] Processing CreateTablet for tablet e94895331d6b474484d987d42f13fd88 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=f51464bcf46d406a807b68e0429a75af]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:08.282259 27740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e94895331d6b474484d987d42f13fd88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:08.282372 27739 tablet_service.cc:1467] Processing CreateTablet for tablet 489db424748b4a73966b047e81f36c83 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=f51464bcf46d406a807b68e0429a75af]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:08.282804 27739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 489db424748b4a73966b047e81f36c83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:08.287039 30784 tablet_service.cc:1467] Processing CreateTablet for tablet e94895331d6b474484d987d42f13fd88 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=f51464bcf46d406a807b68e0429a75af]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:08.287629 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e94895331d6b474484d987d42f13fd88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:08.288113 30517 tablet_service.cc:1467] Processing CreateTablet for tablet 489db424748b4a73966b047e81f36c83 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=f51464bcf46d406a807b68e0429a75af]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:08.288674 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 489db424748b4a73966b047e81f36c83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:08.289865 31374 tablet_bootstrap.cc:492] T c832fea33e4744868b69e6c29e4aea9f P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:08.290196 31373 tablet_bootstrap.cc:492] T c832fea33e4744868b69e6c29e4aea9f P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:08.290292 30514 tablet_service.cc:1467] Processing CreateTablet for tablet e94895331d6b474484d987d42f13fd88 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=f51464bcf46d406a807b68e0429a75af]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:08.290665 30514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e94895331d6b474484d987d42f13fd88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:08.292065 31374 tablet_bootstrap.cc:654] T c832fea33e4744868b69e6c29e4aea9f P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:08.294445 28132 tablet_service.cc:1467] Processing CreateTablet for tablet 489db424748b4a73966b047e81f36c83 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=f51464bcf46d406a807b68e0429a75af]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:08.294785 28132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 489db424748b4a73966b047e81f36c83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:08.294993 31374 tablet_bootstrap.cc:492] T c832fea33e4744868b69e6c29e4aea9f P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:12:08.295130 31374 ts_tablet_manager.cc:1367] T c832fea33e4744868b69e6c29e4aea9f P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 08:12:08.295966 31374 raft_consensus.cc:348] T c832fea33e4744868b69e6c29e4aea9f P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:08.296175 31374 raft_consensus.cc:374] T c832fea33e4744868b69e6c29e4aea9f P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:08.296258 31374 raft_consensus.cc:729] T c832fea33e4744868b69e6c29e4aea9f P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:12:08.296489 31374 consensus_queue.cc:260] T c832fea33e4744868b69e6c29e4aea9f P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:08.296990 31374 ts_tablet_manager.cc:1398] T c832fea33e4744868b69e6c29e4aea9f P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:08.299597 27620 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:08.313565 31358 tablet.cc:1722] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:08.315861 31359 tablet.cc:1722] T 8aae027f329e43d28ccec6be9d1be772 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:08.318400 31373 tablet_bootstrap.cc:654] T c832fea33e4744868b69e6c29e4aea9f P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:08.327461 31360 tablet.cc:1722] T 8aae027f329e43d28ccec6be9d1be772 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:08.333488 27613 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table_new_name [id=5a7febcd91ce4b69992c49762a3b4954] alter complete (version 3)
I20240501 08:12:08.356871 31379 tablet_bootstrap.cc:492] T c832fea33e4744868b69e6c29e4aea9f P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:08.360090 31379 tablet_bootstrap.cc:654] T c832fea33e4744868b69e6c29e4aea9f P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:08.363677 31381 tablet_bootstrap.cc:492] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:08.365231 31380 tablet_bootstrap.cc:492] T 489db424748b4a73966b047e81f36c83 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:08.369444 31377 tablet_bootstrap.cc:492] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:08.372699 31373 tablet_bootstrap.cc:492] T c832fea33e4744868b69e6c29e4aea9f P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:08.372908 31373 ts_tablet_manager.cc:1367] T c832fea33e4744868b69e6c29e4aea9f P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.083s user 0.030s sys 0.012s
I20240501 08:12:08.373574 31381 tablet_bootstrap.cc:654] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:08.373839 31373 raft_consensus.cc:348] T c832fea33e4744868b69e6c29e4aea9f P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:08.374125 31373 raft_consensus.cc:374] T c832fea33e4744868b69e6c29e4aea9f P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:08.374053 31380 tablet_bootstrap.cc:654] T 489db424748b4a73966b047e81f36c83 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:08.374218 31373 raft_consensus.cc:729] T c832fea33e4744868b69e6c29e4aea9f P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:08.374619 31373 consensus_queue.cc:260] T c832fea33e4744868b69e6c29e4aea9f P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:08.375249 31373 ts_tablet_manager.cc:1398] T c832fea33e4744868b69e6c29e4aea9f P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:08.376418 31377 tablet_bootstrap.cc:654] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:08.376996 31373 tablet_bootstrap.cc:492] T e94895331d6b474484d987d42f13fd88 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:08.378262 31379 tablet_bootstrap.cc:492] T c832fea33e4744868b69e6c29e4aea9f P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:08.378262 31380 tablet_bootstrap.cc:492] T 489db424748b4a73966b047e81f36c83 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:08.378683 31380 ts_tablet_manager.cc:1367] T 489db424748b4a73966b047e81f36c83 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.002s
I20240501 08:12:08.378821 31379 ts_tablet_manager.cc:1367] T c832fea33e4744868b69e6c29e4aea9f P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.008s
I20240501 08:12:08.379700 31379 raft_consensus.cc:348] T c832fea33e4744868b69e6c29e4aea9f P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:08.380012 31379 raft_consensus.cc:374] T c832fea33e4744868b69e6c29e4aea9f P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:08.380451 31381 tablet_bootstrap.cc:492] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:12:08.380750 31379 raft_consensus.cc:729] T c832fea33e4744868b69e6c29e4aea9f P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:08.381122 31379 consensus_queue.cc:260] T c832fea33e4744868b69e6c29e4aea9f P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:08.381675 31380 raft_consensus.cc:348] T 489db424748b4a73966b047e81f36c83 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:08.380707 31381 ts_tablet_manager.cc:1367] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20240501 08:12:08.382707 31381 raft_consensus.cc:348] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:08.382851 31379 ts_tablet_manager.cc:1398] T c832fea33e4744868b69e6c29e4aea9f P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 08:12:08.382977 31381 raft_consensus.cc:374] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:08.381891 31380 raft_consensus.cc:374] T 489db424748b4a73966b047e81f36c83 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:08.383150 31381 raft_consensus.cc:729] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:12:08.383623 31380 raft_consensus.cc:729] T 489db424748b4a73966b047e81f36c83 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:08.383916 31381 consensus_queue.cc:260] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:08.384449 31381 ts_tablet_manager.cc:1398] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:08.384673 31380 consensus_queue.cc:260] T 489db424748b4a73966b047e81f36c83 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:08.385309 31380 ts_tablet_manager.cc:1398] T 489db424748b4a73966b047e81f36c83 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 08:12:08.385599 31380 tablet_bootstrap.cc:492] T e94895331d6b474484d987d42f13fd88 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:08.387091 31380 tablet_bootstrap.cc:654] T e94895331d6b474484d987d42f13fd88 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:08.389969 31373 tablet_bootstrap.cc:654] T e94895331d6b474484d987d42f13fd88 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:08.392863 31373 tablet_bootstrap.cc:492] T e94895331d6b474484d987d42f13fd88 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:08.393062 31373 ts_tablet_manager.cc:1367] T e94895331d6b474484d987d42f13fd88 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.003s
I20240501 08:12:08.393846 31373 raft_consensus.cc:348] T e94895331d6b474484d987d42f13fd88 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:08.394110 31373 raft_consensus.cc:374] T e94895331d6b474484d987d42f13fd88 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:08.394191 31373 raft_consensus.cc:729] T e94895331d6b474484d987d42f13fd88 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:08.394416 31373 consensus_queue.cc:260] T e94895331d6b474484d987d42f13fd88 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:08.394871 31373 ts_tablet_manager.cc:1398] T e94895331d6b474484d987d42f13fd88 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:08.407259 31377 tablet_bootstrap.cc:492] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:12:08.407418 31377 ts_tablet_manager.cc:1367] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.000s
I20240501 08:12:08.408068 31377 raft_consensus.cc:348] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:08.408255 31377 raft_consensus.cc:374] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:08.408365 31377 raft_consensus.cc:729] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:12:08.408587 31377 consensus_queue.cc:260] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:08.409086 31377 ts_tablet_manager.cc:1398] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:08.409579 31377 tablet_bootstrap.cc:492] T 489db424748b4a73966b047e81f36c83 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:08.409818 31380 tablet_bootstrap.cc:492] T e94895331d6b474484d987d42f13fd88 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:08.410075 31380 ts_tablet_manager.cc:1367] T e94895331d6b474484d987d42f13fd88 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20240501 08:12:08.410960 31380 raft_consensus.cc:348] T e94895331d6b474484d987d42f13fd88 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:08.411253 31380 raft_consensus.cc:374] T e94895331d6b474484d987d42f13fd88 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:08.411386 31380 raft_consensus.cc:729] T e94895331d6b474484d987d42f13fd88 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:08.411425 31377 tablet_bootstrap.cc:654] T 489db424748b4a73966b047e81f36c83 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:08.411782 31380 consensus_queue.cc:260] T e94895331d6b474484d987d42f13fd88 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:08.416112 31380 ts_tablet_manager.cc:1398] T e94895331d6b474484d987d42f13fd88 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:12:08.421756 31377 tablet_bootstrap.cc:492] T 489db424748b4a73966b047e81f36c83 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:12:08.421962 31377 ts_tablet_manager.cc:1367] T 489db424748b4a73966b047e81f36c83 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.009s
I20240501 08:12:08.422751 31377 raft_consensus.cc:348] T 489db424748b4a73966b047e81f36c83 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:08.423027 31377 raft_consensus.cc:374] T 489db424748b4a73966b047e81f36c83 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:08.423120 31377 raft_consensus.cc:729] T 489db424748b4a73966b047e81f36c83 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:12:08.423375 31377 consensus_queue.cc:260] T 489db424748b4a73966b047e81f36c83 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:08.423907 31377 ts_tablet_manager.cc:1398] T 489db424748b4a73966b047e81f36c83 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:12:08.440192 27617 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:08.440420 27617 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:08.443379 27617 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 8aae027f329e43d28ccec6be9d1be772
I20240501 08:12:08.444329 30517 tablet_service.cc:1514] Processing DeleteTablet for tablet 8aae027f329e43d28ccec6be9d1be772 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:08 UTC) from {username='slave'} at 127.0.0.1:45078
I20240501 08:12:08.444363 30652 tablet_service.cc:1514] Processing DeleteTablet for tablet 8aae027f329e43d28ccec6be9d1be772 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:08 UTC) from {username='slave'} at 127.0.0.1:57156
I20240501 08:12:08.444741 30784 tablet_service.cc:1514] Processing DeleteTablet for tablet 8aae027f329e43d28ccec6be9d1be772 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:08 UTC) from {username='slave'} at 127.0.0.1:38952
I20240501 08:12:08.444988 31390 tablet_replica.cc:330] T 8aae027f329e43d28ccec6be9d1be772 P 31811708c6774037b82df44024b229b6: stopping tablet replica
I20240501 08:12:08.445264 31390 raft_consensus.cc:2229] T 8aae027f329e43d28ccec6be9d1be772 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:08.445313 31392 tablet_replica.cc:330] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5: stopping tablet replica
I20240501 08:12:08.445498 31392 raft_consensus.cc:2229] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:08.445505 31390 raft_consensus.cc:2258] T 8aae027f329e43d28ccec6be9d1be772 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:08.446084 31392 raft_consensus.cc:2258] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:08.447216 31392 ts_tablet_manager.cc:1875] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:08.447343 31390 ts_tablet_manager.cc:1875] T 8aae027f329e43d28ccec6be9d1be772 P 31811708c6774037b82df44024b229b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:08.450342 31391 tablet_replica.cc:330] T 8aae027f329e43d28ccec6be9d1be772 P 9a022e1a501f4d44a0468054d6c3de47: stopping tablet replica
I20240501 08:12:08.450563 31391 raft_consensus.cc:2229] T 8aae027f329e43d28ccec6be9d1be772 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:08.450791 31391 raft_consensus.cc:2258] T 8aae027f329e43d28ccec6be9d1be772 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:08.451611 31391 ts_tablet_manager.cc:1875] T 8aae027f329e43d28ccec6be9d1be772 P 9a022e1a501f4d44a0468054d6c3de47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:08.452926 31392 ts_tablet_manager.cc:1888] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:08.453140 31392 log.cc:1196] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal/wals/8aae027f329e43d28ccec6be9d1be772
I20240501 08:12:08.453675 31392 ts_tablet_manager.cc:1909] T 8aae027f329e43d28ccec6be9d1be772 P 64998323a1f4482d8704bf20594d13a5: Deleting consensus metadata
I20240501 08:12:08.454283 31390 ts_tablet_manager.cc:1888] T 8aae027f329e43d28ccec6be9d1be772 P 31811708c6774037b82df44024b229b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:08.454411 31390 log.cc:1196] T 8aae027f329e43d28ccec6be9d1be772 P 31811708c6774037b82df44024b229b6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/wals/8aae027f329e43d28ccec6be9d1be772
I20240501 08:12:08.454872 31390 ts_tablet_manager.cc:1909] T 8aae027f329e43d28ccec6be9d1be772 P 31811708c6774037b82df44024b229b6: Deleting consensus metadata
I20240501 08:12:08.454880 27605 catalog_manager.cc:4830] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): tablet 8aae027f329e43d28ccec6be9d1be772 (table rebalancer_dynamic_table_new_name [id=5a7febcd91ce4b69992c49762a3b4954]) successfully deleted
I20240501 08:12:08.455863 27607 catalog_manager.cc:4830] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): tablet 8aae027f329e43d28ccec6be9d1be772 (table rebalancer_dynamic_table_new_name [id=5a7febcd91ce4b69992c49762a3b4954]) successfully deleted
I20240501 08:12:08.458479 31391 ts_tablet_manager.cc:1888] T 8aae027f329e43d28ccec6be9d1be772 P 9a022e1a501f4d44a0468054d6c3de47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:08.458598 31391 log.cc:1196] T 8aae027f329e43d28ccec6be9d1be772 P 9a022e1a501f4d44a0468054d6c3de47: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/wals/8aae027f329e43d28ccec6be9d1be772
I20240501 08:12:08.458952 31391 ts_tablet_manager.cc:1909] T 8aae027f329e43d28ccec6be9d1be772 P 9a022e1a501f4d44a0468054d6c3de47: Deleting consensus metadata
I20240501 08:12:08.459877 27604 catalog_manager.cc:4830] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): tablet 8aae027f329e43d28ccec6be9d1be772 (table rebalancer_dynamic_table_new_name [id=5a7febcd91ce4b69992c49762a3b4954]) successfully deleted
I20240501 08:12:08.500388 30854 raft_consensus.cc:482] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:08.500631 30854 raft_consensus.cc:504] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:08.501435 30854 leader_election.cc:288] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:08.502027 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "489db424748b4a73966b047e81f36c83" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:12:08.502276 27761 raft_consensus.cc:2454] T 489db424748b4a73966b047e81f36c83 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:12:08.502874 28087 leader_election.cc:302] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:12:08.503211 30854 raft_consensus.cc:2781] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:08.503348 30854 raft_consensus.cc:482] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:08.503464 30854 raft_consensus.cc:3037] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:08.506614 30854 raft_consensus.cc:504] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:08.507534 30854 leader_election.cc:288] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:08.507887 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "489db424748b4a73966b047e81f36c83" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:12:08.508144 27761 raft_consensus.cc:3037] T 489db424748b4a73966b047e81f36c83 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:08.509207 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "489db424748b4a73966b047e81f36c83" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:08.509289 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "489db424748b4a73966b047e81f36c83" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:08.509511 30536 raft_consensus.cc:2454] T 489db424748b4a73966b047e81f36c83 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:12:08.510649 27761 raft_consensus.cc:2454] T 489db424748b4a73966b047e81f36c83 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:12:08.511029 28087 leader_election.cc:302] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters: 31811708c6774037b82df44024b229b6
I20240501 08:12:08.511339 30854 raft_consensus.cc:2781] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:08.511467 30854 raft_consensus.cc:686] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:12:08.511700 30854 consensus_queue.cc:237] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:08.514990 27617 catalog_manager.cc:5484] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:08.545743 27617 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:08.552018 31176 raft_consensus.cc:482] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:08.552223 31176 raft_consensus.cc:504] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:08.553042 31176 leader_election.cc:288] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:08.553709 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e94895331d6b474484d987d42f13fd88" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:08.554008 30536 raft_consensus.cc:2454] T e94895331d6b474484d987d42f13fd88 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:12:08.553967 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e94895331d6b474484d987d42f13fd88" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:08.554226 30804 raft_consensus.cc:2454] T e94895331d6b474484d987d42f13fd88 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:12:08.554442 30784 tablet_service.cc:1467] Processing CreateTablet for tablet dd1e69effa5e40028ee7ac9063483358 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dbff374d87a146f9b9433e96788d179c]), partition=
I20240501 08:12:08.554680 27695 leader_election.cc:302] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 64998323a1f4482d8704bf20594d13a5; no voters:
I20240501 08:12:08.554793 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd1e69effa5e40028ee7ac9063483358. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:08.554843 30916 tablet_service.cc:1467] Processing CreateTablet for tablet dd1e69effa5e40028ee7ac9063483358 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dbff374d87a146f9b9433e96788d179c]), partition=
I20240501 08:12:08.554986 30517 tablet_service.cc:1467] Processing CreateTablet for tablet dd1e69effa5e40028ee7ac9063483358 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dbff374d87a146f9b9433e96788d179c]), partition=
I20240501 08:12:08.555205 30916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd1e69effa5e40028ee7ac9063483358. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:08.555224 31176 raft_consensus.cc:2781] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:08.555300 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd1e69effa5e40028ee7ac9063483358. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:08.555330 31176 raft_consensus.cc:482] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:08.555408 31176 raft_consensus.cc:3037] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:08.558203 31176 raft_consensus.cc:504] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:08.559630 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e94895331d6b474484d987d42f13fd88" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:08.559885 30804 raft_consensus.cc:3037] T e94895331d6b474484d987d42f13fd88 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:08.561503 31379 tablet_bootstrap.cc:492] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:08.563169 31379 tablet_bootstrap.cc:654] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:08.563482 30804 raft_consensus.cc:2454] T e94895331d6b474484d987d42f13fd88 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:12:08.563926 27695 leader_election.cc:302] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 64998323a1f4482d8704bf20594d13a5; no voters:
I20240501 08:12:08.564282 31176 leader_election.cc:288] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Requested vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:08.564428 31362 raft_consensus.cc:2781] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:08.564857 31362 raft_consensus.cc:686] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:12:08.564967 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e94895331d6b474484d987d42f13fd88" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:08.565213 30536 raft_consensus.cc:3037] T e94895331d6b474484d987d42f13fd88 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:08.565186 31362 consensus_queue.cc:237] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:08.567826 31380 tablet_bootstrap.cc:492] T dd1e69effa5e40028ee7ac9063483358 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:08.568256 30536 raft_consensus.cc:2454] T e94895331d6b474484d987d42f13fd88 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:12:08.569473 31380 tablet_bootstrap.cc:654] T dd1e69effa5e40028ee7ac9063483358 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:08.570495 31373 tablet_bootstrap.cc:492] T dd1e69effa5e40028ee7ac9063483358 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:08.571494 27620 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240501 08:12:08.571720 27620 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240501 08:12:08.571869 31373 tablet_bootstrap.cc:654] T dd1e69effa5e40028ee7ac9063483358 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:08.572831 27617 catalog_manager.cc:5484] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:08.575528 31379 tablet_bootstrap.cc:492] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:08.575709 31379 ts_tablet_manager.cc:1367] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240501 08:12:08.576440 31379 raft_consensus.cc:348] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:08.576644 31379 raft_consensus.cc:374] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:08.576709 31379 raft_consensus.cc:729] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:08.576959 31379 consensus_queue.cc:260] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:08.579136 27620 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 271d8e2e56d2468b8f0f9b7835c74a62
I20240501 08:12:08.579706 27620 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 188e377fef614ce1801a8f9306d789e1
I20240501 08:12:08.580070 27620 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 428edf53f79e4b6d9f4b455cba16e644
I20240501 08:12:08.580044 27871 tablet_service.cc:1514] Processing DeleteTablet for tablet 271d8e2e56d2468b8f0f9b7835c74a62 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:08 UTC) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:08.580348 31223 tablet_replica.cc:330] T 271d8e2e56d2468b8f0f9b7835c74a62 P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:08.580556 31223 raft_consensus.cc:2229] T 271d8e2e56d2468b8f0f9b7835c74a62 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:08.580857 31223 raft_consensus.cc:2258] T 271d8e2e56d2468b8f0f9b7835c74a62 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:08.581379 27871 tablet_service.cc:1514] Processing DeleteTablet for tablet 188e377fef614ce1801a8f9306d789e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:08 UTC) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:08.581593 31223 ts_tablet_manager.cc:1875] T 271d8e2e56d2468b8f0f9b7835c74a62 P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:08.584448 28132 tablet_service.cc:1514] Processing DeleteTablet for tablet 188e377fef614ce1801a8f9306d789e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:08 UTC) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:08.584784 31031 tablet_replica.cc:330] T 188e377fef614ce1801a8f9306d789e1 P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:08.585095 31031 raft_consensus.cc:2229] T 188e377fef614ce1801a8f9306d789e1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:08.585605 31031 raft_consensus.cc:2258] T 188e377fef614ce1801a8f9306d789e1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:08.586721 28000 tablet_service.cc:1514] Processing DeleteTablet for tablet 271d8e2e56d2468b8f0f9b7835c74a62 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:08 UTC) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:08.586982 28002 tablet_service.cc:1514] Processing DeleteTablet for tablet 428edf53f79e4b6d9f4b455cba16e644 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:08 UTC) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:08.587198 31323 tablet_replica.cc:330] T 271d8e2e56d2468b8f0f9b7835c74a62 P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:08.587404 31323 raft_consensus.cc:2229] T 271d8e2e56d2468b8f0f9b7835c74a62 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:08.587415 31031 ts_tablet_manager.cc:1875] T 188e377fef614ce1801a8f9306d789e1 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:08.587630 31323 raft_consensus.cc:2258] T 271d8e2e56d2468b8f0f9b7835c74a62 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:08.588322 31323 ts_tablet_manager.cc:1875] T 271d8e2e56d2468b8f0f9b7835c74a62 P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:08.591404 27740 tablet_service.cc:1514] Processing DeleteTablet for tablet 271d8e2e56d2468b8f0f9b7835c74a62 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:08 UTC) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:08.591552 27741 tablet_service.cc:1514] Processing DeleteTablet for tablet 428edf53f79e4b6d9f4b455cba16e644 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:08 UTC) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:08.591686 31032 tablet_replica.cc:330] T 271d8e2e56d2468b8f0f9b7835c74a62 P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:08.591897 31032 raft_consensus.cc:2229] T 271d8e2e56d2468b8f0f9b7835c74a62 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:08.592178 31032 raft_consensus.cc:2258] T 271d8e2e56d2468b8f0f9b7835c74a62 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:08.592876 31032 ts_tablet_manager.cc:1875] T 271d8e2e56d2468b8f0f9b7835c74a62 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:08.596464 31323 ts_tablet_manager.cc:1888] T 271d8e2e56d2468b8f0f9b7835c74a62 P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:08.596573 31323 log.cc:1196] T 271d8e2e56d2468b8f0f9b7835c74a62 P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/271d8e2e56d2468b8f0f9b7835c74a62
I20240501 08:12:08.596874 31323 ts_tablet_manager.cc:1909] T 271d8e2e56d2468b8f0f9b7835c74a62 P aad81047009a4ffcbc450b60030583b5: Deleting consensus metadata
I20240501 08:12:08.597689 31323 tablet_replica.cc:330] T 428edf53f79e4b6d9f4b455cba16e644 P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:08.597815 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet 271d8e2e56d2468b8f0f9b7835c74a62 (table rebalance_test_table_16 [id=5e12abb0d0d6480c8906c3ec242309fc]) successfully deleted
I20240501 08:12:08.597851 31031 ts_tablet_manager.cc:1888] T 188e377fef614ce1801a8f9306d789e1 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:08.598044 31031 log.cc:1196] T 188e377fef614ce1801a8f9306d789e1 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/188e377fef614ce1801a8f9306d789e1
I20240501 08:12:08.591404 27739 tablet_service.cc:1514] Processing DeleteTablet for tablet 188e377fef614ce1801a8f9306d789e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:08 UTC) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:08.598417 31379 ts_tablet_manager.cc:1398] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.023s user 0.001s sys 0.000s
I20240501 08:12:08.598475 31031 ts_tablet_manager.cc:1909] T 188e377fef614ce1801a8f9306d789e1 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting consensus metadata
I20240501 08:12:08.599320 31008 raft_consensus.cc:482] T c832fea33e4744868b69e6c29e4aea9f P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:08.599521 31008 raft_consensus.cc:504] T c832fea33e4744868b69e6c29e4aea9f P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:08.600394 31008 leader_election.cc:288] T c832fea33e4744868b69e6c29e4aea9f P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:08.600971 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet 188e377fef614ce1801a8f9306d789e1 (table rebalance_test_table_16 [id=5e12abb0d0d6480c8906c3ec242309fc]) successfully deleted
I20240501 08:12:08.601150 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c832fea33e4744868b69e6c29e4aea9f" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:08.601429 30804 raft_consensus.cc:2454] T c832fea33e4744868b69e6c29e4aea9f P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:08.601552 28132 tablet_service.cc:1514] Processing DeleteTablet for tablet 428edf53f79e4b6d9f4b455cba16e644 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:08 UTC) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:08.601832 31031 tablet_replica.cc:330] T 428edf53f79e4b6d9f4b455cba16e644 P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:08.602059 31031 raft_consensus.cc:2229] T 428edf53f79e4b6d9f4b455cba16e644 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:08.602154 31032 ts_tablet_manager.cc:1888] T 271d8e2e56d2468b8f0f9b7835c74a62 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:08.602290 31032 log.cc:1196] T 271d8e2e56d2468b8f0f9b7835c74a62 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/271d8e2e56d2468b8f0f9b7835c74a62
I20240501 08:12:08.602420 31031 raft_consensus.cc:2258] T 428edf53f79e4b6d9f4b455cba16e644 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:08.602895 30871 leader_election.cc:302] T c832fea33e4744868b69e6c29e4aea9f P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:08.602986 31032 ts_tablet_manager.cc:1909] T 271d8e2e56d2468b8f0f9b7835c74a62 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting consensus metadata
I20240501 08:12:08.603191 31031 ts_tablet_manager.cc:1875] T 428edf53f79e4b6d9f4b455cba16e644 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:08.603745 31223 ts_tablet_manager.cc:1888] T 271d8e2e56d2468b8f0f9b7835c74a62 P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:08.603940 31223 log.cc:1196] T 271d8e2e56d2468b8f0f9b7835c74a62 P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/271d8e2e56d2468b8f0f9b7835c74a62
I20240501 08:12:08.603928 31032 tablet_replica.cc:330] T 428edf53f79e4b6d9f4b455cba16e644 P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:08.604161 31032 raft_consensus.cc:2229] T 428edf53f79e4b6d9f4b455cba16e644 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:08.604382 31032 raft_consensus.cc:2258] T 428edf53f79e4b6d9f4b455cba16e644 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:08.604406 31223 ts_tablet_manager.cc:1909] T 271d8e2e56d2468b8f0f9b7835c74a62 P f1525cbc82a24d13b65f136695c903af: Deleting consensus metadata
I20240501 08:12:08.605012 31032 ts_tablet_manager.cc:1875] T 428edf53f79e4b6d9f4b455cba16e644 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:08.605445 31223 tablet_replica.cc:330] T 188e377fef614ce1801a8f9306d789e1 P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:08.605530 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet 271d8e2e56d2468b8f0f9b7835c74a62 (table rebalance_test_table_16 [id=5e12abb0d0d6480c8906c3ec242309fc]) successfully deleted
I20240501 08:12:08.605655 31223 raft_consensus.cc:2229] T 188e377fef614ce1801a8f9306d789e1 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:08.605657 31008 raft_consensus.cc:2781] T c832fea33e4744868b69e6c29e4aea9f P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:08.605756 31008 raft_consensus.cc:482] T c832fea33e4744868b69e6c29e4aea9f P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:08.605832 31008 raft_consensus.cc:3037] T c832fea33e4744868b69e6c29e4aea9f P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:08.597892 31323 raft_consensus.cc:2229] T 428edf53f79e4b6d9f4b455cba16e644 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:08.608081 31323 raft_consensus.cc:2258] T 428edf53f79e4b6d9f4b455cba16e644 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:08.608886 31323 ts_tablet_manager.cc:1875] T 428edf53f79e4b6d9f4b455cba16e644 P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:08.610093 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet 271d8e2e56d2468b8f0f9b7835c74a62 (table rebalance_test_table_16 [id=5e12abb0d0d6480c8906c3ec242309fc]) successfully deleted
I20240501 08:12:08.611320 31031 ts_tablet_manager.cc:1888] T 428edf53f79e4b6d9f4b455cba16e644 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:08.611456 31031 log.cc:1196] T 428edf53f79e4b6d9f4b455cba16e644 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/428edf53f79e4b6d9f4b455cba16e644
I20240501 08:12:08.611884 31031 ts_tablet_manager.cc:1909] T 428edf53f79e4b6d9f4b455cba16e644 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting consensus metadata
I20240501 08:12:08.612994 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet 428edf53f79e4b6d9f4b455cba16e644 (table rebalance_test_table_16 [id=5e12abb0d0d6480c8906c3ec242309fc]) successfully deleted
I20240501 08:12:08.614671 31380 tablet_bootstrap.cc:492] T dd1e69effa5e40028ee7ac9063483358 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:08.614878 31380 ts_tablet_manager.cc:1367] T dd1e69effa5e40028ee7ac9063483358 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.047s user 0.001s sys 0.003s
I20240501 08:12:08.605912 31223 raft_consensus.cc:2258] T 188e377fef614ce1801a8f9306d789e1 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:08.615896 31380 raft_consensus.cc:348] T dd1e69effa5e40028ee7ac9063483358 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:08.616585 31223 ts_tablet_manager.cc:1875] T 188e377fef614ce1801a8f9306d789e1 P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:08.617406 31032 ts_tablet_manager.cc:1888] T 428edf53f79e4b6d9f4b455cba16e644 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:08.617532 31032 log.cc:1196] T 428edf53f79e4b6d9f4b455cba16e644 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/428edf53f79e4b6d9f4b455cba16e644
I20240501 08:12:08.618016 31032 ts_tablet_manager.cc:1909] T 428edf53f79e4b6d9f4b455cba16e644 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting consensus metadata
I20240501 08:12:08.618546 28019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c832fea33e4744868b69e6c29e4aea9f" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:12:08.618808 28019 raft_consensus.cc:2454] T c832fea33e4744868b69e6c29e4aea9f P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:08.619174 31032 tablet_replica.cc:330] T 188e377fef614ce1801a8f9306d789e1 P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:08.619383 31032 raft_consensus.cc:2229] T 188e377fef614ce1801a8f9306d789e1 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:08.619511 31373 tablet_bootstrap.cc:492] T dd1e69effa5e40028ee7ac9063483358 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:08.619611 31032 raft_consensus.cc:2258] T 188e377fef614ce1801a8f9306d789e1 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:08.619659 31373 ts_tablet_manager.cc:1367] T dd1e69effa5e40028ee7ac9063483358 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.049s user 0.008s sys 0.000s
I20240501 08:12:08.619738 31008 raft_consensus.cc:504] T c832fea33e4744868b69e6c29e4aea9f P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:08.620349 31032 ts_tablet_manager.cc:1875] T 188e377fef614ce1801a8f9306d789e1 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:08.620486 31373 raft_consensus.cc:348] T dd1e69effa5e40028ee7ac9063483358 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:08.620759 31373 raft_consensus.cc:374] T dd1e69effa5e40028ee7ac9063483358 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:08.620858 31373 raft_consensus.cc:729] T dd1e69effa5e40028ee7ac9063483358 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:08.621093 31373 consensus_queue.cc:260] T dd1e69effa5e40028ee7ac9063483358 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:08.621330 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c832fea33e4744868b69e6c29e4aea9f" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:08.621546 30804 raft_consensus.cc:3037] T c832fea33e4744868b69e6c29e4aea9f P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:08.621884 31373 ts_tablet_manager.cc:1398] T dd1e69effa5e40028ee7ac9063483358 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:08.622990 31223 ts_tablet_manager.cc:1888] T 188e377fef614ce1801a8f9306d789e1 P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:08.623138 31223 log.cc:1196] T 188e377fef614ce1801a8f9306d789e1 P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/188e377fef614ce1801a8f9306d789e1
I20240501 08:12:08.623551 31223 ts_tablet_manager.cc:1909] T 188e377fef614ce1801a8f9306d789e1 P f1525cbc82a24d13b65f136695c903af: Deleting consensus metadata
I20240501 08:12:08.624667 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet 188e377fef614ce1801a8f9306d789e1 (table rebalance_test_table_16 [id=5e12abb0d0d6480c8906c3ec242309fc]) successfully deleted
I20240501 08:12:08.626283 31323 ts_tablet_manager.cc:1888] T 428edf53f79e4b6d9f4b455cba16e644 P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:08.626480 31323 log.cc:1196] T 428edf53f79e4b6d9f4b455cba16e644 P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/428edf53f79e4b6d9f4b455cba16e644
I20240501 08:12:08.626758 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet 428edf53f79e4b6d9f4b455cba16e644 (table rebalance_test_table_16 [id=5e12abb0d0d6480c8906c3ec242309fc]) successfully deleted
I20240501 08:12:08.627102 31323 ts_tablet_manager.cc:1909] T 428edf53f79e4b6d9f4b455cba16e644 P aad81047009a4ffcbc450b60030583b5: Deleting consensus metadata
I20240501 08:12:08.627305 30804 raft_consensus.cc:2454] T c832fea33e4744868b69e6c29e4aea9f P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:08.627446 31380 raft_consensus.cc:374] T dd1e69effa5e40028ee7ac9063483358 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:08.627565 31380 raft_consensus.cc:729] T dd1e69effa5e40028ee7ac9063483358 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:08.627810 30871 leader_election.cc:302] T c832fea33e4744868b69e6c29e4aea9f P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:08.627996 31008 leader_election.cc:288] T c832fea33e4744868b69e6c29e4aea9f P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Requested vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:08.628211 31008 raft_consensus.cc:2781] T c832fea33e4744868b69e6c29e4aea9f P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:08.628335 31008 raft_consensus.cc:686] T c832fea33e4744868b69e6c29e4aea9f P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Becoming Leader. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:12:08.628613 31008 consensus_queue.cc:237] T c832fea33e4744868b69e6c29e4aea9f P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:08.628749 28019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c832fea33e4744868b69e6c29e4aea9f" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:12:08.628978 28019 raft_consensus.cc:3037] T c832fea33e4744868b69e6c29e4aea9f P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:08.630072 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet 428edf53f79e4b6d9f4b455cba16e644 (table rebalance_test_table_16 [id=5e12abb0d0d6480c8906c3ec242309fc]) successfully deleted
I20240501 08:12:08.630568 31380 consensus_queue.cc:260] T dd1e69effa5e40028ee7ac9063483358 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:08.631363 31380 ts_tablet_manager.cc:1398] T dd1e69effa5e40028ee7ac9063483358 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.016s user 0.000s sys 0.003s
I20240501 08:12:08.632615 28019 raft_consensus.cc:2454] T c832fea33e4744868b69e6c29e4aea9f P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:08.634287 31032 ts_tablet_manager.cc:1888] T 188e377fef614ce1801a8f9306d789e1 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:08.634542 31032 log.cc:1196] T 188e377fef614ce1801a8f9306d789e1 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/188e377fef614ce1801a8f9306d789e1
I20240501 08:12:08.635128 31032 ts_tablet_manager.cc:1909] T 188e377fef614ce1801a8f9306d789e1 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting consensus metadata
I20240501 08:12:08.636291 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet 188e377fef614ce1801a8f9306d789e1 (table rebalance_test_table_16 [id=5e12abb0d0d6480c8906c3ec242309fc]) successfully deleted
I20240501 08:12:08.640291 27617 catalog_manager.cc:5484] T c832fea33e4744868b69e6c29e4aea9f P a29cd590999b4f32849ff5689cd53c11 reported cstate change: term changed from 0 to 1, leader changed from <none> to a29cd590999b4f32849ff5689cd53c11 (127.21.84.136). New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:08.677973 31008 raft_consensus.cc:482] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:08.678314 31008 raft_consensus.cc:504] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:08.679050 31008 leader_election.cc:288] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:08.679476 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd1e69effa5e40028ee7ac9063483358" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:08.679607 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd1e69effa5e40028ee7ac9063483358" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:08.679847 30536 raft_consensus.cc:2454] T dd1e69effa5e40028ee7ac9063483358 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:08.679843 30804 raft_consensus.cc:2454] T dd1e69effa5e40028ee7ac9063483358 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:08.680233 30873 leader_election.cc:302] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:08.680544 31008 raft_consensus.cc:2781] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:08.680652 31008 raft_consensus.cc:482] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:08.680754 31008 raft_consensus.cc:3037] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:08.682776 31008 raft_consensus.cc:504] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:08.683346 31008 leader_election.cc:288] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Requested vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:08.683780 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd1e69effa5e40028ee7ac9063483358" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:08.683804 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd1e69effa5e40028ee7ac9063483358" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:08.683961 30536 raft_consensus.cc:3037] T dd1e69effa5e40028ee7ac9063483358 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:08.683961 30803 raft_consensus.cc:3037] T dd1e69effa5e40028ee7ac9063483358 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:08.685858 30536 raft_consensus.cc:2454] T dd1e69effa5e40028ee7ac9063483358 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:08.685901 30803 raft_consensus.cc:2454] T dd1e69effa5e40028ee7ac9063483358 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:08.686358 30873 leader_election.cc:302] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:08.686630 31008 raft_consensus.cc:2781] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:08.686827 31008 raft_consensus.cc:686] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Becoming Leader. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:12:08.687088 31008 consensus_queue.cc:237] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:08.690163 27617 catalog_manager.cc:5484] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: term changed from 0 to 1, leader changed from <none> to a29cd590999b4f32849ff5689cd53c11 (127.21.84.136). New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:08.810628 27620 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:08.816567 30536 raft_consensus.cc:1261] T dd1e69effa5e40028ee7ac9063483358 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:08.816629 30803 raft_consensus.cc:1261] T dd1e69effa5e40028ee7ac9063483358 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:08.817531 31403 consensus_queue.cc:1035] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:08.818468 31008 consensus_queue.cc:1035] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:08.831862 27620 catalog_manager.cc:5576] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293) does not have the latest schema for tablet dd1e69effa5e40028ee7ac9063483358 (table rebalancer_dynamic_table [id=dbff374d87a146f9b9433e96788d179c]). Expected version 1 got 0
I20240501 08:12:08.832355 31415 tablet.cc:1722] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:08.832602 27617 catalog_manager.cc:5576] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503) does not have the latest schema for tablet dd1e69effa5e40028ee7ac9063483358 (table rebalancer_dynamic_table [id=dbff374d87a146f9b9433e96788d179c]). Expected version 1 got 0
I20240501 08:12:08.833119 31415 tablet.cc:1620] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11: MemRowSet was empty: no flush needed.
I20240501 08:12:08.834813 31409 mvcc.cc:205] Tried to move back new op lower bound from 7022801423627079680 to 7022801423105048576. Current Snapshot: MvccSnapshot[applied={T|T < 7022801423627079680}]
I20240501 08:12:08.838500 27617 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=dbff374d87a146f9b9433e96788d179c] alter complete (version 1)
I20240501 08:12:08.839569 31416 tablet.cc:1722] T dd1e69effa5e40028ee7ac9063483358 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:08.841869 27620 catalog_manager.cc:5576] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503) does not have the latest schema for tablet dd1e69effa5e40028ee7ac9063483358 (table rebalancer_dynamic_table [id=dbff374d87a146f9b9433e96788d179c]). Expected version 1 got 0
I20240501 08:12:08.842254 31416 tablet.cc:1620] T dd1e69effa5e40028ee7ac9063483358 P 64998323a1f4482d8704bf20594d13a5: MemRowSet was empty: no flush needed.
I20240501 08:12:08.844875 31417 tablet.cc:1722] T dd1e69effa5e40028ee7ac9063483358 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:08.845459 31417 tablet.cc:1620] T dd1e69effa5e40028ee7ac9063483358 P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:08.845461 27620 catalog_manager.cc:5576] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293) does not have the latest schema for tablet dd1e69effa5e40028ee7ac9063483358 (table rebalancer_dynamic_table [id=dbff374d87a146f9b9433e96788d179c]). Expected version 1 got 0
I20240501 08:12:08.906242 30854 consensus_queue.cc:1035] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:08.910605 30536 raft_consensus.cc:3037] T 489db424748b4a73966b047e81f36c83 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:08.930920 31420 consensus_queue.cc:1035] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:08.945685 27620 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:08.953020 31415 tablet.cc:1722] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:08.953562 31415 tablet.cc:1620] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11: MemRowSet was empty: no flush needed.
I20240501 08:12:08.953519 31416 tablet.cc:1722] T dd1e69effa5e40028ee7ac9063483358 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:08.954010 31416 tablet.cc:1620] T dd1e69effa5e40028ee7ac9063483358 P 64998323a1f4482d8704bf20594d13a5: MemRowSet was empty: no flush needed.
I20240501 08:12:08.954102 31417 tablet.cc:1722] T dd1e69effa5e40028ee7ac9063483358 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:08.954519 31417 tablet.cc:1620] T dd1e69effa5e40028ee7ac9063483358 P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:08.965782 27620 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=dbff374d87a146f9b9433e96788d179c] alter complete (version 2)
I20240501 08:12:09.065198 31403 consensus_queue.cc:1035] T c832fea33e4744868b69e6c29e4aea9f P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:09.069801 31414 consensus_queue.cc:1035] T c832fea33e4744868b69e6c29e4aea9f P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:09.071506 27620 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:09.078745 31415 tablet.cc:1722] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:09.079548 31416 tablet.cc:1722] T dd1e69effa5e40028ee7ac9063483358 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:09.085448 31417 tablet.cc:1722] T dd1e69effa5e40028ee7ac9063483358 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:09.090677 27620 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table_new_name [id=dbff374d87a146f9b9433e96788d179c] alter complete (version 3)
I20240501 08:12:09.095525 27617 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240501 08:12:09.096227 27617 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240501 08:12:09.098583 31362 consensus_queue.cc:1035] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:09.103528 27617 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet a87f31b95dda450d8c0cb122a175a9a5
I20240501 08:12:09.104418 28132 tablet_service.cc:1514] Processing DeleteTablet for tablet a87f31b95dda450d8c0cb122a175a9a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:09 UTC) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:09.104779 27871 tablet_service.cc:1514] Processing DeleteTablet for tablet a87f31b95dda450d8c0cb122a175a9a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:09 UTC) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:09.104964 27739 tablet_service.cc:1514] Processing DeleteTablet for tablet a87f31b95dda450d8c0cb122a175a9a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:09 UTC) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:09.105074 31223 tablet_replica.cc:330] T a87f31b95dda450d8c0cb122a175a9a5 P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:09.105090 31031 tablet_replica.cc:330] T a87f31b95dda450d8c0cb122a175a9a5 P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:09.105229 31032 tablet_replica.cc:330] T a87f31b95dda450d8c0cb122a175a9a5 P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:09.105316 31223 raft_consensus.cc:2229] T a87f31b95dda450d8c0cb122a175a9a5 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:09.105319 31031 raft_consensus.cc:2229] T a87f31b95dda450d8c0cb122a175a9a5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:09.105427 31032 raft_consensus.cc:2229] T a87f31b95dda450d8c0cb122a175a9a5 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:09.105563 31223 raft_consensus.cc:2258] T a87f31b95dda450d8c0cb122a175a9a5 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:09.105664 31031 raft_consensus.cc:2258] T a87f31b95dda450d8c0cb122a175a9a5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:09.105664 31032 raft_consensus.cc:2258] T a87f31b95dda450d8c0cb122a175a9a5 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:09.106348 31223 ts_tablet_manager.cc:1875] T a87f31b95dda450d8c0cb122a175a9a5 P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:09.106405 31032 ts_tablet_manager.cc:1875] T a87f31b95dda450d8c0cb122a175a9a5 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:09.106418 31031 ts_tablet_manager.cc:1875] T a87f31b95dda450d8c0cb122a175a9a5 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:09.113056 31223 ts_tablet_manager.cc:1888] T a87f31b95dda450d8c0cb122a175a9a5 P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:09.113181 31031 ts_tablet_manager.cc:1888] T a87f31b95dda450d8c0cb122a175a9a5 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:09.113355 31031 log.cc:1196] T a87f31b95dda450d8c0cb122a175a9a5 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/a87f31b95dda450d8c0cb122a175a9a5
I20240501 08:12:09.113373 31223 log.cc:1196] T a87f31b95dda450d8c0cb122a175a9a5 P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/a87f31b95dda450d8c0cb122a175a9a5
I20240501 08:12:09.114180 31223 ts_tablet_manager.cc:1909] T a87f31b95dda450d8c0cb122a175a9a5 P f1525cbc82a24d13b65f136695c903af: Deleting consensus metadata
I20240501 08:12:09.114367 31031 ts_tablet_manager.cc:1909] T a87f31b95dda450d8c0cb122a175a9a5 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting consensus metadata
I20240501 08:12:09.115401 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet a87f31b95dda450d8c0cb122a175a9a5 (table rebalance_test_table_18 [id=c750e4f2dd5641909963ec3f64f43082]) successfully deleted
I20240501 08:12:09.115465 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet a87f31b95dda450d8c0cb122a175a9a5 (table rebalance_test_table_18 [id=c750e4f2dd5641909963ec3f64f43082]) successfully deleted
I20240501 08:12:09.114681 27617 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 59a15f7f0dd646908578446fa5532a25
I20240501 08:12:09.121671 31362 consensus_queue.cc:1035] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:09.123003 31032 ts_tablet_manager.cc:1888] T a87f31b95dda450d8c0cb122a175a9a5 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:09.123139 31032 log.cc:1196] T a87f31b95dda450d8c0cb122a175a9a5 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/a87f31b95dda450d8c0cb122a175a9a5
I20240501 08:12:09.123618 31032 ts_tablet_manager.cc:1909] T a87f31b95dda450d8c0cb122a175a9a5 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting consensus metadata
I20240501 08:12:09.124688 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet a87f31b95dda450d8c0cb122a175a9a5 (table rebalance_test_table_18 [id=c750e4f2dd5641909963ec3f64f43082]) successfully deleted
I20240501 08:12:09.126560 27739 tablet_service.cc:1514] Processing DeleteTablet for tablet 59a15f7f0dd646908578446fa5532a25 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:09 UTC) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:09.126922 28002 tablet_service.cc:1514] Processing DeleteTablet for tablet 59a15f7f0dd646908578446fa5532a25 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:09 UTC) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:09.127077 27617 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 7fc24ac81fdd48e6baf42e3e675e76d9
I20240501 08:12:09.127480 31323 tablet_replica.cc:330] T 59a15f7f0dd646908578446fa5532a25 P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:09.127683 28131 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fc24ac81fdd48e6baf42e3e675e76d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:09 UTC) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:09.127736 31323 raft_consensus.cc:2229] T 59a15f7f0dd646908578446fa5532a25 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:09.127971 31323 raft_consensus.cc:2258] T 59a15f7f0dd646908578446fa5532a25 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:09.128255 27871 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fc24ac81fdd48e6baf42e3e675e76d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:09 UTC) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:09.128504 31223 tablet_replica.cc:330] T 7fc24ac81fdd48e6baf42e3e675e76d9 P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:09.128708 31223 raft_consensus.cc:2229] T 7fc24ac81fdd48e6baf42e3e675e76d9 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:09.128748 31323 ts_tablet_manager.cc:1875] T 59a15f7f0dd646908578446fa5532a25 P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:09.128928 31223 raft_consensus.cc:2258] T 7fc24ac81fdd48e6baf42e3e675e76d9 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:09.129474 31223 ts_tablet_manager.cc:1875] T 7fc24ac81fdd48e6baf42e3e675e76d9 P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:09.130097 28132 tablet_service.cc:1514] Processing DeleteTablet for tablet 59a15f7f0dd646908578446fa5532a25 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:09 UTC) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:09.131093 27739 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fc24ac81fdd48e6baf42e3e675e76d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:09 UTC) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:09.131155 31032 tablet_replica.cc:330] T 59a15f7f0dd646908578446fa5532a25 P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:09.132985 31032 raft_consensus.cc:2229] T 59a15f7f0dd646908578446fa5532a25 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:09.133402 31032 raft_consensus.cc:2258] T 59a15f7f0dd646908578446fa5532a25 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:09.134263 31032 ts_tablet_manager.cc:1875] T 59a15f7f0dd646908578446fa5532a25 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:09.136721 31223 ts_tablet_manager.cc:1888] T 7fc24ac81fdd48e6baf42e3e675e76d9 P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:09.136921 31223 log.cc:1196] T 7fc24ac81fdd48e6baf42e3e675e76d9 P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/7fc24ac81fdd48e6baf42e3e675e76d9
I20240501 08:12:09.137485 31223 ts_tablet_manager.cc:1909] T 7fc24ac81fdd48e6baf42e3e675e76d9 P f1525cbc82a24d13b65f136695c903af: Deleting consensus metadata
I20240501 08:12:09.138703 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet 7fc24ac81fdd48e6baf42e3e675e76d9 (table rebalance_test_table_18 [id=c750e4f2dd5641909963ec3f64f43082]) successfully deleted
I20240501 08:12:09.140039 31323 ts_tablet_manager.cc:1888] T 59a15f7f0dd646908578446fa5532a25 P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:09.140203 31323 log.cc:1196] T 59a15f7f0dd646908578446fa5532a25 P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/59a15f7f0dd646908578446fa5532a25
I20240501 08:12:09.140650 31323 ts_tablet_manager.cc:1909] T 59a15f7f0dd646908578446fa5532a25 P aad81047009a4ffcbc450b60030583b5: Deleting consensus metadata
I20240501 08:12:09.141758 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet 59a15f7f0dd646908578446fa5532a25 (table rebalance_test_table_18 [id=c750e4f2dd5641909963ec3f64f43082]) successfully deleted
I20240501 08:12:09.142165 31031 tablet_replica.cc:330] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:09.142287 31032 ts_tablet_manager.cc:1888] T 59a15f7f0dd646908578446fa5532a25 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:09.142407 31032 log.cc:1196] T 59a15f7f0dd646908578446fa5532a25 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/59a15f7f0dd646908578446fa5532a25
I20240501 08:12:09.142372 31031 raft_consensus.cc:2229] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:09.142603 31031 raft_consensus.cc:2258] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:09.142716 31032 ts_tablet_manager.cc:1909] T 59a15f7f0dd646908578446fa5532a25 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting consensus metadata
I20240501 08:12:09.143229 31031 ts_tablet_manager.cc:1875] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:09.143540 31032 tablet_replica.cc:330] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:09.143752 31032 raft_consensus.cc:2229] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:09.144217 31032 raft_consensus.cc:2258] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:09.144225 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet 59a15f7f0dd646908578446fa5532a25 (table rebalance_test_table_18 [id=c750e4f2dd5641909963ec3f64f43082]) successfully deleted
I20240501 08:12:09.144824 31032 ts_tablet_manager.cc:1875] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:09.148155 31031 ts_tablet_manager.cc:1888] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:09.148275 31031 log.cc:1196] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/7fc24ac81fdd48e6baf42e3e675e76d9
I20240501 08:12:09.148625 31031 ts_tablet_manager.cc:1909] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting consensus metadata
I20240501 08:12:09.149052 31032 ts_tablet_manager.cc:1888] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:09.149155 31032 log.cc:1196] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/7fc24ac81fdd48e6baf42e3e675e76d9
I20240501 08:12:09.149451 31031 tablet_replica.cc:330] T 59a15f7f0dd646908578446fa5532a25 P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:09.149608 31032 ts_tablet_manager.cc:1909] T 7fc24ac81fdd48e6baf42e3e675e76d9 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting consensus metadata
I20240501 08:12:09.149668 31031 raft_consensus.cc:2229] T 59a15f7f0dd646908578446fa5532a25 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:09.149617 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet 7fc24ac81fdd48e6baf42e3e675e76d9 (table rebalance_test_table_18 [id=c750e4f2dd5641909963ec3f64f43082]) successfully deleted
I20240501 08:12:09.149888 31031 raft_consensus.cc:2258] T 59a15f7f0dd646908578446fa5532a25 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:09.150552 31031 ts_tablet_manager.cc:1875] T 59a15f7f0dd646908578446fa5532a25 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:09.150661 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet 7fc24ac81fdd48e6baf42e3e675e76d9 (table rebalance_test_table_18 [id=c750e4f2dd5641909963ec3f64f43082]) successfully deleted
I20240501 08:12:09.155488 31031 ts_tablet_manager.cc:1888] T 59a15f7f0dd646908578446fa5532a25 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:09.155630 31031 log.cc:1196] T 59a15f7f0dd646908578446fa5532a25 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/59a15f7f0dd646908578446fa5532a25
I20240501 08:12:09.155961 31031 ts_tablet_manager.cc:1909] T 59a15f7f0dd646908578446fa5532a25 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting consensus metadata
I20240501 08:12:09.156852 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet 59a15f7f0dd646908578446fa5532a25 (table rebalance_test_table_18 [id=c750e4f2dd5641909963ec3f64f43082]) successfully deleted
I20240501 08:12:09.194023 27617 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:09.194293 27617 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:09.197206 27617 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet dd1e69effa5e40028ee7ac9063483358
I20240501 08:12:09.198313 30517 tablet_service.cc:1514] Processing DeleteTablet for tablet dd1e69effa5e40028ee7ac9063483358 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:09 UTC) from {username='slave'} at 127.0.0.1:45078
I20240501 08:12:09.198308 30784 tablet_service.cc:1514] Processing DeleteTablet for tablet dd1e69effa5e40028ee7ac9063483358 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:09 UTC) from {username='slave'} at 127.0.0.1:38952
I20240501 08:12:09.198825 30916 tablet_service.cc:1514] Processing DeleteTablet for tablet dd1e69effa5e40028ee7ac9063483358 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:09 UTC) from {username='slave'} at 127.0.0.1:54036
I20240501 08:12:09.198885 31430 tablet_replica.cc:330] T dd1e69effa5e40028ee7ac9063483358 P 31811708c6774037b82df44024b229b6: stopping tablet replica
I20240501 08:12:09.199074 31430 raft_consensus.cc:2229] T dd1e69effa5e40028ee7ac9063483358 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:09.199271 31430 raft_consensus.cc:2258] T dd1e69effa5e40028ee7ac9063483358 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:09.200310 31429 tablet_replica.cc:330] T dd1e69effa5e40028ee7ac9063483358 P 64998323a1f4482d8704bf20594d13a5: stopping tablet replica
I20240501 08:12:09.200461 31430 ts_tablet_manager.cc:1875] T dd1e69effa5e40028ee7ac9063483358 P 31811708c6774037b82df44024b229b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:09.200661 31429 raft_consensus.cc:2229] T dd1e69effa5e40028ee7ac9063483358 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:09.201025 31429 raft_consensus.cc:2258] T dd1e69effa5e40028ee7ac9063483358 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:09.202051 31429 ts_tablet_manager.cc:1875] T dd1e69effa5e40028ee7ac9063483358 P 64998323a1f4482d8704bf20594d13a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:09.202385 31431 tablet_replica.cc:330] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11: stopping tablet replica
I20240501 08:12:09.202637 31431 raft_consensus.cc:2229] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:09.203186 31431 raft_consensus.cc:2258] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:09.204020 31431 ts_tablet_manager.cc:1875] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:09.206547 31430 ts_tablet_manager.cc:1888] T dd1e69effa5e40028ee7ac9063483358 P 31811708c6774037b82df44024b229b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:09.206730 31430 log.cc:1196] T dd1e69effa5e40028ee7ac9063483358 P 31811708c6774037b82df44024b229b6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/wals/dd1e69effa5e40028ee7ac9063483358
I20240501 08:12:09.207262 31430 ts_tablet_manager.cc:1909] T dd1e69effa5e40028ee7ac9063483358 P 31811708c6774037b82df44024b229b6: Deleting consensus metadata
I20240501 08:12:09.207648 27617 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_extra_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:09.208720 27607 catalog_manager.cc:4830] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): tablet dd1e69effa5e40028ee7ac9063483358 (table rebalancer_dynamic_table_new_name [id=dbff374d87a146f9b9433e96788d179c]) successfully deleted
I20240501 08:12:09.209092 31429 ts_tablet_manager.cc:1888] T dd1e69effa5e40028ee7ac9063483358 P 64998323a1f4482d8704bf20594d13a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:09.209246 31429 log.cc:1196] T dd1e69effa5e40028ee7ac9063483358 P 64998323a1f4482d8704bf20594d13a5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal/wals/dd1e69effa5e40028ee7ac9063483358
I20240501 08:12:09.209772 31429 ts_tablet_manager.cc:1909] T dd1e69effa5e40028ee7ac9063483358 P 64998323a1f4482d8704bf20594d13a5: Deleting consensus metadata
I20240501 08:12:09.210865 27605 catalog_manager.cc:4830] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): tablet dd1e69effa5e40028ee7ac9063483358 (table rebalancer_dynamic_table_new_name [id=dbff374d87a146f9b9433e96788d179c]) successfully deleted
I20240501 08:12:09.211025 31431 ts_tablet_manager.cc:1888] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:09.211165 31431 log.cc:1196] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/wals/dd1e69effa5e40028ee7ac9063483358
I20240501 08:12:09.211551 31431 ts_tablet_manager.cc:1909] T dd1e69effa5e40028ee7ac9063483358 P a29cd590999b4f32849ff5689cd53c11: Deleting consensus metadata
I20240501 08:12:09.212760 27606 catalog_manager.cc:4830] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): tablet dd1e69effa5e40028ee7ac9063483358 (table rebalancer_dynamic_table_new_name [id=dbff374d87a146f9b9433e96788d179c]) successfully deleted
I20240501 08:12:09.217005 27739 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab0410a98974002ab68dd0f6a456d59 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=6377dbf65f184f9cbe44f12bd6a0405f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:09.217516 27739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab0410a98974002ab68dd0f6a456d59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:09.217526 28132 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab0410a98974002ab68dd0f6a456d59 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=6377dbf65f184f9cbe44f12bd6a0405f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:09.217675 30517 tablet_service.cc:1467] Processing CreateTablet for tablet 1448b8f2d5844ad7aec9bcd3874c5c01 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=6377dbf65f184f9cbe44f12bd6a0405f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:09.218020 28132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab0410a98974002ab68dd0f6a456d59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:09.218112 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1448b8f2d5844ad7aec9bcd3874c5c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:09.222337 30916 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab0410a98974002ab68dd0f6a456d59 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=6377dbf65f184f9cbe44f12bd6a0405f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:09.222998 30916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab0410a98974002ab68dd0f6a456d59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:09.225284 30917 tablet_service.cc:1467] Processing CreateTablet for tablet 1448b8f2d5844ad7aec9bcd3874c5c01 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=6377dbf65f184f9cbe44f12bd6a0405f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:09.225853 30917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1448b8f2d5844ad7aec9bcd3874c5c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:09.231953 27741 tablet_service.cc:1467] Processing CreateTablet for tablet 103e1394bfe34a86ba1d9487d75fd512 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=6377dbf65f184f9cbe44f12bd6a0405f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:09.232026 30514 tablet_service.cc:1467] Processing CreateTablet for tablet 103e1394bfe34a86ba1d9487d75fd512 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=6377dbf65f184f9cbe44f12bd6a0405f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:09.232461 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 103e1394bfe34a86ba1d9487d75fd512. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:09.232473 30514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 103e1394bfe34a86ba1d9487d75fd512. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:09.235901 30652 tablet_service.cc:1467] Processing CreateTablet for tablet 1448b8f2d5844ad7aec9bcd3874c5c01 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=6377dbf65f184f9cbe44f12bd6a0405f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:09.236366 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1448b8f2d5844ad7aec9bcd3874c5c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:09.238438 30651 tablet_service.cc:1467] Processing CreateTablet for tablet 103e1394bfe34a86ba1d9487d75fd512 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=6377dbf65f184f9cbe44f12bd6a0405f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:09.238834 30651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 103e1394bfe34a86ba1d9487d75fd512. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:09.246697 31435 tablet_bootstrap.cc:492] T 8ab0410a98974002ab68dd0f6a456d59 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:09.246855 31432 tablet_bootstrap.cc:492] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:09.248700 31432 tablet_bootstrap.cc:654] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:09.250514 31434 tablet_bootstrap.cc:492] T 8ab0410a98974002ab68dd0f6a456d59 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:09.252395 31435 tablet_bootstrap.cc:654] T 8ab0410a98974002ab68dd0f6a456d59 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:09.254287 31434 tablet_bootstrap.cc:654] T 8ab0410a98974002ab68dd0f6a456d59 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:09.263204 31432 tablet_bootstrap.cc:492] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:12:09.263514 31432 ts_tablet_manager.cc:1367] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.009s
I20240501 08:12:09.264576 31432 raft_consensus.cc:348] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:09.265070 31432 raft_consensus.cc:374] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:09.265239 31432 raft_consensus.cc:729] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:12:09.265718 31439 tablet_bootstrap.cc:492] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:09.265601 31432 consensus_queue.cc:260] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:09.267254 31432 ts_tablet_manager.cc:1398] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:09.267611 31432 tablet_bootstrap.cc:492] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:09.270880 31434 tablet_bootstrap.cc:492] T 8ab0410a98974002ab68dd0f6a456d59 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:09.271101 31432 tablet_bootstrap.cc:654] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:09.273155 31434 ts_tablet_manager.cc:1367] T 8ab0410a98974002ab68dd0f6a456d59 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.000s
I20240501 08:12:09.274544 31434 raft_consensus.cc:348] T 8ab0410a98974002ab68dd0f6a456d59 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:09.275825 31439 tablet_bootstrap.cc:654] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:09.276924 31434 raft_consensus.cc:374] T 8ab0410a98974002ab68dd0f6a456d59 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:09.277055 31434 raft_consensus.cc:729] T 8ab0410a98974002ab68dd0f6a456d59 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:09.277487 31435 tablet_bootstrap.cc:492] T 8ab0410a98974002ab68dd0f6a456d59 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:12:09.277377 31434 consensus_queue.cc:260] T 8ab0410a98974002ab68dd0f6a456d59 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:09.277762 31435 ts_tablet_manager.cc:1367] T 8ab0410a98974002ab68dd0f6a456d59 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.014s
I20240501 08:12:09.278048 31434 ts_tablet_manager.cc:1398] T 8ab0410a98974002ab68dd0f6a456d59 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:12:09.278404 31434 tablet_bootstrap.cc:492] T 1448b8f2d5844ad7aec9bcd3874c5c01 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:09.278674 31435 raft_consensus.cc:348] T 8ab0410a98974002ab68dd0f6a456d59 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:09.279002 31435 raft_consensus.cc:374] T 8ab0410a98974002ab68dd0f6a456d59 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:09.279153 31435 raft_consensus.cc:729] T 8ab0410a98974002ab68dd0f6a456d59 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:12:09.279470 31435 consensus_queue.cc:260] T 8ab0410a98974002ab68dd0f6a456d59 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:09.280529 31435 ts_tablet_manager.cc:1398] T 8ab0410a98974002ab68dd0f6a456d59 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:09.280596 31432 tablet_bootstrap.cc:492] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:12:09.280850 31432 ts_tablet_manager.cc:1367] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.007s
I20240501 08:12:09.280905 31433 tablet_bootstrap.cc:492] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:09.281132 31434 tablet_bootstrap.cc:654] T 1448b8f2d5844ad7aec9bcd3874c5c01 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:09.282779 31432 raft_consensus.cc:348] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:09.283078 31432 raft_consensus.cc:374] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:09.283206 31432 raft_consensus.cc:729] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:12:09.283552 31432 consensus_queue.cc:260] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:09.283921 31433 tablet_bootstrap.cc:654] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:09.285087 31432 ts_tablet_manager.cc:1398] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:09.286556 31433 tablet_bootstrap.cc:492] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:09.286788 31433 ts_tablet_manager.cc:1367] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 08:12:09.288189 31433 raft_consensus.cc:348] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:09.288482 31433 raft_consensus.cc:374] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:09.288630 31433 raft_consensus.cc:729] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:09.288970 31433 consensus_queue.cc:260] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:09.289652 31433 ts_tablet_manager.cc:1398] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:09.290108 31433 tablet_bootstrap.cc:492] T 103e1394bfe34a86ba1d9487d75fd512 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:09.291967 31433 tablet_bootstrap.cc:654] T 103e1394bfe34a86ba1d9487d75fd512 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:09.302110 31433 tablet_bootstrap.cc:492] T 103e1394bfe34a86ba1d9487d75fd512 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:09.302395 31433 ts_tablet_manager.cc:1367] T 103e1394bfe34a86ba1d9487d75fd512 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20240501 08:12:09.303853 31433 raft_consensus.cc:348] T 103e1394bfe34a86ba1d9487d75fd512 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:09.304198 31433 raft_consensus.cc:374] T 103e1394bfe34a86ba1d9487d75fd512 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:09.304322 31433 raft_consensus.cc:729] T 103e1394bfe34a86ba1d9487d75fd512 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:09.304625 31433 consensus_queue.cc:260] T 103e1394bfe34a86ba1d9487d75fd512 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:09.301492 27613 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:09.305413 31433 ts_tablet_manager.cc:1398] T 103e1394bfe34a86ba1d9487d75fd512 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:09.307353 31362 raft_consensus.cc:482] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:09.307543 31362 raft_consensus.cc:504] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:09.308454 31362 leader_election.cc:288] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:09.309023 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab0410a98974002ab68dd0f6a456d59" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:12:09.309307 28150 raft_consensus.cc:2454] T 8ab0410a98974002ab68dd0f6a456d59 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:12:09.309862 27694 leader_election.cc:302] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:12:09.310322 31362 raft_consensus.cc:2781] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:09.310441 31362 raft_consensus.cc:482] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:09.310560 31362 raft_consensus.cc:3037] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:09.321789 31292 raft_consensus.cc:482] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:09.323115 31292 raft_consensus.cc:504] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:09.323988 31292 leader_election.cc:288] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:09.325103 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "103e1394bfe34a86ba1d9487d75fd512" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:09.325423 30536 raft_consensus.cc:2454] T 103e1394bfe34a86ba1d9487d75fd512 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:12:09.326095 27697 leader_election.cc:302] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 6331f77dae9d452184e8cd0cbf5f1b43; no voters:
I20240501 08:12:09.329178 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "103e1394bfe34a86ba1d9487d75fd512" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:09.330169 30784 tablet_service.cc:1467] Processing CreateTablet for tablet 5b3f83f48de44cc7bec327afaff68953 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e730f17f1ded41b79e871afc5cacb78a]), partition=
W20240501 08:12:09.330332 27694 leader_election.cc:341] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:09.330716 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b3f83f48de44cc7bec327afaff68953. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:09.330879 31292 raft_consensus.cc:2781] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:09.330996 31292 raft_consensus.cc:482] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:09.331113 31292 raft_consensus.cc:3037] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:09.335242 31292 raft_consensus.cc:504] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:09.336856 31362 raft_consensus.cc:504] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:09.338326 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab0410a98974002ab68dd0f6a456d59" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:12:09.338634 28150 raft_consensus.cc:3037] T 8ab0410a98974002ab68dd0f6a456d59 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:09.339939 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "103e1394bfe34a86ba1d9487d75fd512" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
W20240501 08:12:09.340654 27694 leader_election.cc:341] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:09.343045 31362 leader_election.cc:288] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Requested vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:09.343422 31292 leader_election.cc:288] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Requested vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:09.348196 31439 tablet_bootstrap.cc:492] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:09.348425 31439 ts_tablet_manager.cc:1367] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.083s user 0.004s sys 0.000s
I20240501 08:12:09.349341 30652 tablet_service.cc:1467] Processing CreateTablet for tablet 5b3f83f48de44cc7bec327afaff68953 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e730f17f1ded41b79e871afc5cacb78a]), partition=
I20240501 08:12:09.349550 31439 raft_consensus.cc:348] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:09.349802 31439 raft_consensus.cc:374] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:09.349897 31439 raft_consensus.cc:729] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:09.349969 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b3f83f48de44cc7bec327afaff68953. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:09.350194 31439 consensus_queue.cc:260] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:09.350996 31439 ts_tablet_manager.cc:1398] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:12:09.351326 31439 tablet_bootstrap.cc:492] T 103e1394bfe34a86ba1d9487d75fd512 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:09.353195 31439 tablet_bootstrap.cc:654] T 103e1394bfe34a86ba1d9487d75fd512 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:09.356009 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "103e1394bfe34a86ba1d9487d75fd512" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:09.356273 30536 raft_consensus.cc:3037] T 103e1394bfe34a86ba1d9487d75fd512 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:09.358394 30536 raft_consensus.cc:2454] T 103e1394bfe34a86ba1d9487d75fd512 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:12:09.358991 27697 leader_election.cc:302] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31811708c6774037b82df44024b229b6, 6331f77dae9d452184e8cd0cbf5f1b43; no voters: 9a022e1a501f4d44a0468054d6c3de47
I20240501 08:12:09.359330 31434 tablet_bootstrap.cc:492] T 1448b8f2d5844ad7aec9bcd3874c5c01 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:09.359485 31362 raft_consensus.cc:2781] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:09.359546 31434 ts_tablet_manager.cc:1367] T 1448b8f2d5844ad7aec9bcd3874c5c01 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.081s user 0.016s sys 0.048s
I20240501 08:12:09.359679 31362 raft_consensus.cc:686] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:12:09.359944 31362 consensus_queue.cc:237] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:09.360426 31434 raft_consensus.cc:348] T 1448b8f2d5844ad7aec9bcd3874c5c01 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:09.360982 31434 raft_consensus.cc:374] T 1448b8f2d5844ad7aec9bcd3874c5c01 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:09.361340 31434 raft_consensus.cc:729] T 1448b8f2d5844ad7aec9bcd3874c5c01 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:09.361892 30517 tablet_service.cc:1467] Processing CreateTablet for tablet 5b3f83f48de44cc7bec327afaff68953 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e730f17f1ded41b79e871afc5cacb78a]), partition=
I20240501 08:12:09.362303 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b3f83f48de44cc7bec327afaff68953. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:09.361969 31434 consensus_queue.cc:260] T 1448b8f2d5844ad7aec9bcd3874c5c01 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:09.364454 31434 ts_tablet_manager.cc:1398] T 1448b8f2d5844ad7aec9bcd3874c5c01 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240501 08:12:09.365767 30937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab0410a98974002ab68dd0f6a456d59" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:09.365809 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab0410a98974002ab68dd0f6a456d59" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:09.366225 30937 raft_consensus.cc:2454] T 8ab0410a98974002ab68dd0f6a456d59 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:12:09.393185 31433 tablet_bootstrap.cc:492] T 5b3f83f48de44cc7bec327afaff68953 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:09.394912 31433 tablet_bootstrap.cc:654] T 5b3f83f48de44cc7bec327afaff68953 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:09.397720 28150 raft_consensus.cc:2454] T 8ab0410a98974002ab68dd0f6a456d59 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:12:09.410926 31433 tablet_bootstrap.cc:492] T 5b3f83f48de44cc7bec327afaff68953 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:09.410869 27694 leader_election.cc:302] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters: a29cd590999b4f32849ff5689cd53c11
I20240501 08:12:09.411190 31433 ts_tablet_manager.cc:1367] T 5b3f83f48de44cc7bec327afaff68953 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.005s
I20240501 08:12:09.411396 31292 raft_consensus.cc:2781] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:09.411581 31292 raft_consensus.cc:686] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:12:09.411895 31292 consensus_queue.cc:237] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:09.412331 31433 raft_consensus.cc:348] T 5b3f83f48de44cc7bec327afaff68953 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:09.412636 31433 raft_consensus.cc:374] T 5b3f83f48de44cc7bec327afaff68953 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:09.412782 31433 raft_consensus.cc:729] T 5b3f83f48de44cc7bec327afaff68953 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:09.413106 31433 consensus_queue.cc:260] T 5b3f83f48de44cc7bec327afaff68953 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:09.414834 31433 ts_tablet_manager.cc:1398] T 5b3f83f48de44cc7bec327afaff68953 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:09.435532 31451 tablet_bootstrap.cc:492] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:09.437049 31451 tablet_bootstrap.cc:654] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:09.440863 27620 catalog_manager.cc:5484] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:09.442138 31439 tablet_bootstrap.cc:492] T 103e1394bfe34a86ba1d9487d75fd512 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:09.442389 31439 ts_tablet_manager.cc:1367] T 103e1394bfe34a86ba1d9487d75fd512 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.091s user 0.022s sys 0.032s
I20240501 08:12:09.443325 31451 tablet_bootstrap.cc:492] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:09.443480 31451 ts_tablet_manager.cc:1367] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240501 08:12:09.444146 31451 raft_consensus.cc:348] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:09.444453 31451 raft_consensus.cc:374] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:09.444603 31451 raft_consensus.cc:729] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:09.444504 31439 raft_consensus.cc:348] T 103e1394bfe34a86ba1d9487d75fd512 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:09.444736 31439 raft_consensus.cc:374] T 103e1394bfe34a86ba1d9487d75fd512 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:09.444869 31439 raft_consensus.cc:729] T 103e1394bfe34a86ba1d9487d75fd512 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:09.445196 31439 consensus_queue.cc:260] T 103e1394bfe34a86ba1d9487d75fd512 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:09.445065 31451 consensus_queue.cc:260] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:09.446148 31451 ts_tablet_manager.cc:1398] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:09.446538 31439 ts_tablet_manager.cc:1398] T 103e1394bfe34a86ba1d9487d75fd512 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:09.449728 27613 catalog_manager.cc:5484] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:09.446867 31439 tablet_bootstrap.cc:492] T 5b3f83f48de44cc7bec327afaff68953 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:09.455536 31439 tablet_bootstrap.cc:654] T 5b3f83f48de44cc7bec327afaff68953 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:09.458685 31439 tablet_bootstrap.cc:492] T 5b3f83f48de44cc7bec327afaff68953 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:09.458848 31439 ts_tablet_manager.cc:1367] T 5b3f83f48de44cc7bec327afaff68953 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240501 08:12:09.459475 31439 raft_consensus.cc:348] T 5b3f83f48de44cc7bec327afaff68953 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:09.459635 31439 raft_consensus.cc:374] T 5b3f83f48de44cc7bec327afaff68953 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:09.459721 31439 raft_consensus.cc:729] T 5b3f83f48de44cc7bec327afaff68953 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:09.459924 31439 consensus_queue.cc:260] T 5b3f83f48de44cc7bec327afaff68953 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:09.460330 31439 ts_tablet_manager.cc:1398] T 5b3f83f48de44cc7bec327afaff68953 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:12:09.568635 31348 raft_consensus.cc:482] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:09.568892 31348 raft_consensus.cc:504] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:09.569626 31348 leader_election.cc:288] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:09.570145 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1448b8f2d5844ad7aec9bcd3874c5c01" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:09.570199 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1448b8f2d5844ad7aec9bcd3874c5c01" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:09.570377 30936 raft_consensus.cc:2454] T 1448b8f2d5844ad7aec9bcd3874c5c01 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 0.
I20240501 08:12:09.570437 30672 raft_consensus.cc:2454] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 0.
I20240501 08:12:09.570813 30472 leader_election.cc:302] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:09.571096 31348 raft_consensus.cc:2781] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:09.571238 31348 raft_consensus.cc:482] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:09.571321 31348 raft_consensus.cc:3037] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:09.573181 31348 raft_consensus.cc:504] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:09.573863 31348 leader_election.cc:288] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 election: Requested vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:09.574417 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1448b8f2d5844ad7aec9bcd3874c5c01" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:09.574656 30936 raft_consensus.cc:3037] T 1448b8f2d5844ad7aec9bcd3874c5c01 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:09.574671 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1448b8f2d5844ad7aec9bcd3874c5c01" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:09.574930 30672 raft_consensus.cc:3037] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:09.577737 30936 raft_consensus.cc:2454] T 1448b8f2d5844ad7aec9bcd3874c5c01 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 1.
I20240501 08:12:09.577796 30672 raft_consensus.cc:2454] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 1.
I20240501 08:12:09.578262 30472 leader_election.cc:302] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:09.578685 31348 raft_consensus.cc:2781] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:09.579166 31348 raft_consensus.cc:686] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Becoming Leader. State: Replica: 31811708c6774037b82df44024b229b6, State: Running, Role: LEADER
I20240501 08:12:09.579499 31348 consensus_queue.cc:237] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:09.582418 27617 catalog_manager.cc:5484] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31811708c6774037b82df44024b229b6 (127.21.84.133). New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:09.583441 31312 raft_consensus.cc:482] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:09.583606 31312 raft_consensus.cc:504] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:09.584939 31312 leader_election.cc:288] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:09.585526 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3f83f48de44cc7bec327afaff68953" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:09.585565 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3f83f48de44cc7bec327afaff68953" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:09.585777 30672 raft_consensus.cc:2454] T 5b3f83f48de44cc7bec327afaff68953 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:09.585776 30536 raft_consensus.cc:2454] T 5b3f83f48de44cc7bec327afaff68953 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:09.586226 30740 leader_election.cc:302] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 64998323a1f4482d8704bf20594d13a5; no voters:
I20240501 08:12:09.586537 31312 raft_consensus.cc:2781] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:09.586644 31312 raft_consensus.cc:482] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:09.586726 31312 raft_consensus.cc:3037] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:09.588526 31312 raft_consensus.cc:504] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:09.589186 31312 leader_election.cc:288] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Requested vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:09.589825 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3f83f48de44cc7bec327afaff68953" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:09.590050 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3f83f48de44cc7bec327afaff68953" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:09.590122 30672 raft_consensus.cc:3037] T 5b3f83f48de44cc7bec327afaff68953 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:09.590250 30536 raft_consensus.cc:3037] T 5b3f83f48de44cc7bec327afaff68953 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:09.592962 30536 raft_consensus.cc:2454] T 5b3f83f48de44cc7bec327afaff68953 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:09.593001 30672 raft_consensus.cc:2454] T 5b3f83f48de44cc7bec327afaff68953 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:09.593411 30737 leader_election.cc:302] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:09.593742 31312 raft_consensus.cc:2781] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:09.594107 31312 raft_consensus.cc:686] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Becoming Leader. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Running, Role: LEADER
I20240501 08:12:09.594759 31312 consensus_queue.cc:237] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:09.597723 27617 catalog_manager.cc:5484] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64998323a1f4482d8704bf20594d13a5 (127.21.84.135). New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:09.739840 27617 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:09.746016 30672 raft_consensus.cc:1261] T 5b3f83f48de44cc7bec327afaff68953 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:09.745682 30536 raft_consensus.cc:1261] T 5b3f83f48de44cc7bec327afaff68953 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:09.746959 31458 consensus_queue.cc:1035] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:09.747287 31312 consensus_queue.cc:1035] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:09.751957 27617 catalog_manager.cc:5576] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933) does not have the latest schema for tablet 5b3f83f48de44cc7bec327afaff68953 (table rebalancer_dynamic_table [id=e730f17f1ded41b79e871afc5cacb78a]). Expected version 1 got 0
I20240501 08:12:09.753855 31465 tablet.cc:1722] T 5b3f83f48de44cc7bec327afaff68953 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:09.754505 31465 tablet.cc:1620] T 5b3f83f48de44cc7bec327afaff68953 P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:09.756021 27620 catalog_manager.cc:5576] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933) does not have the latest schema for tablet 5b3f83f48de44cc7bec327afaff68953 (table rebalancer_dynamic_table [id=e730f17f1ded41b79e871afc5cacb78a]). Expected version 1 got 0
I20240501 08:12:09.759862 31464 tablet.cc:1722] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:09.762619 27617 catalog_manager.cc:5576] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503) does not have the latest schema for tablet 5b3f83f48de44cc7bec327afaff68953 (table rebalancer_dynamic_table [id=e730f17f1ded41b79e871afc5cacb78a]). Expected version 1 got 0
I20240501 08:12:09.766425 31464 tablet.cc:1620] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5: MemRowSet was empty: no flush needed.
I20240501 08:12:09.767565 27617 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=e730f17f1ded41b79e871afc5cacb78a] alter complete (version 1)
I20240501 08:12:09.769192 27620 catalog_manager.cc:5576] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293) does not have the latest schema for tablet 5b3f83f48de44cc7bec327afaff68953 (table rebalancer_dynamic_table [id=e730f17f1ded41b79e871afc5cacb78a]). Expected version 1 got 0
I20240501 08:12:09.771683 31468 tablet.cc:1722] T 5b3f83f48de44cc7bec327afaff68953 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:09.772269 31468 tablet.cc:1620] T 5b3f83f48de44cc7bec327afaff68953 P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:09.773087 30672 raft_consensus.cc:3037] T 103e1394bfe34a86ba1d9487d75fd512 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:09.774472 27620 catalog_manager.cc:5576] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293) does not have the latest schema for tablet 5b3f83f48de44cc7bec327afaff68953 (table rebalancer_dynamic_table [id=e730f17f1ded41b79e871afc5cacb78a]). Expected version 1 got 0
I20240501 08:12:09.777365 31292 consensus_queue.cc:1035] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:09.782698 31362 consensus_queue.cc:1035] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:09.813664 30936 raft_consensus.cc:3037] T 8ab0410a98974002ab68dd0f6a456d59 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:09.816399 31362 consensus_queue.cc:1035] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:09.820530 31176 consensus_queue.cc:1035] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:09.869558 27620 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:09.876829 31464 tablet.cc:1722] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:09.877162 31465 tablet.cc:1722] T 5b3f83f48de44cc7bec327afaff68953 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:09.877453 31464 tablet.cc:1620] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5: MemRowSet was empty: no flush needed.
I20240501 08:12:09.877647 31465 tablet.cc:1620] T 5b3f83f48de44cc7bec327afaff68953 P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:09.877651 31468 tablet.cc:1722] T 5b3f83f48de44cc7bec327afaff68953 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:09.878149 31468 tablet.cc:1620] T 5b3f83f48de44cc7bec327afaff68953 P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:09.885862 27617 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=e730f17f1ded41b79e871afc5cacb78a] alter complete (version 2)
I20240501 08:12:09.991648 27620 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:09.999457 31464 tablet.cc:1722] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:10.001410 31468 tablet.cc:1722] T 5b3f83f48de44cc7bec327afaff68953 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:10.001695 31465 tablet.cc:1722] T 5b3f83f48de44cc7bec327afaff68953 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:10.007359 27613 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table_new_name [id=e730f17f1ded41b79e871afc5cacb78a] alter complete (version 3)
I20240501 08:12:10.110920 27613 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:10.111155 27613 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:10.113837 27613 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 5b3f83f48de44cc7bec327afaff68953
I20240501 08:12:10.114643 30784 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b3f83f48de44cc7bec327afaff68953 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:10 UTC) from {username='slave'} at 127.0.0.1:38952
I20240501 08:12:10.114742 30517 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b3f83f48de44cc7bec327afaff68953 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:10 UTC) from {username='slave'} at 127.0.0.1:45078
I20240501 08:12:10.114969 30652 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b3f83f48de44cc7bec327afaff68953 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:10 UTC) from {username='slave'} at 127.0.0.1:57156
I20240501 08:12:10.116299 31477 tablet_replica.cc:330] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5: stopping tablet replica
I20240501 08:12:10.116437 31480 tablet_replica.cc:330] T 5b3f83f48de44cc7bec327afaff68953 P 9a022e1a501f4d44a0468054d6c3de47: stopping tablet replica
I20240501 08:12:10.116578 31479 tablet_replica.cc:330] T 5b3f83f48de44cc7bec327afaff68953 P 31811708c6774037b82df44024b229b6: stopping tablet replica
I20240501 08:12:10.116613 31477 raft_consensus.cc:2229] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:10.116658 31480 raft_consensus.cc:2229] T 5b3f83f48de44cc7bec327afaff68953 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:10.116899 31480 raft_consensus.cc:2258] T 5b3f83f48de44cc7bec327afaff68953 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:10.116914 31479 raft_consensus.cc:2229] T 5b3f83f48de44cc7bec327afaff68953 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:10.117004 31477 raft_consensus.cc:2258] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:10.117120 31479 raft_consensus.cc:2258] T 5b3f83f48de44cc7bec327afaff68953 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:10.117753 31480 ts_tablet_manager.cc:1875] T 5b3f83f48de44cc7bec327afaff68953 P 9a022e1a501f4d44a0468054d6c3de47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:10.117799 31477 ts_tablet_manager.cc:1875] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:10.118889 31479 ts_tablet_manager.cc:1875] T 5b3f83f48de44cc7bec327afaff68953 P 31811708c6774037b82df44024b229b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:10.122246 31480 ts_tablet_manager.cc:1888] T 5b3f83f48de44cc7bec327afaff68953 P 9a022e1a501f4d44a0468054d6c3de47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:10.122371 31480 log.cc:1196] T 5b3f83f48de44cc7bec327afaff68953 P 9a022e1a501f4d44a0468054d6c3de47: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/wals/5b3f83f48de44cc7bec327afaff68953
I20240501 08:12:10.122804 31480 ts_tablet_manager.cc:1909] T 5b3f83f48de44cc7bec327afaff68953 P 9a022e1a501f4d44a0468054d6c3de47: Deleting consensus metadata
I20240501 08:12:10.123826 27604 catalog_manager.cc:4830] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): tablet 5b3f83f48de44cc7bec327afaff68953 (table rebalancer_dynamic_table_new_name [id=e730f17f1ded41b79e871afc5cacb78a]) successfully deleted
I20240501 08:12:10.124120 31477 ts_tablet_manager.cc:1888] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:10.124274 31477 log.cc:1196] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal/wals/5b3f83f48de44cc7bec327afaff68953
I20240501 08:12:10.124722 31477 ts_tablet_manager.cc:1909] T 5b3f83f48de44cc7bec327afaff68953 P 64998323a1f4482d8704bf20594d13a5: Deleting consensus metadata
I20240501 08:12:10.125270 31479 ts_tablet_manager.cc:1888] T 5b3f83f48de44cc7bec327afaff68953 P 31811708c6774037b82df44024b229b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:10.125401 31479 log.cc:1196] T 5b3f83f48de44cc7bec327afaff68953 P 31811708c6774037b82df44024b229b6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/wals/5b3f83f48de44cc7bec327afaff68953
I20240501 08:12:10.125855 31479 ts_tablet_manager.cc:1909] T 5b3f83f48de44cc7bec327afaff68953 P 31811708c6774037b82df44024b229b6: Deleting consensus metadata
I20240501 08:12:10.125869 27605 catalog_manager.cc:4830] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): tablet 5b3f83f48de44cc7bec327afaff68953 (table rebalancer_dynamic_table_new_name [id=e730f17f1ded41b79e871afc5cacb78a]) successfully deleted
I20240501 08:12:10.126832 27607 catalog_manager.cc:4830] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): tablet 5b3f83f48de44cc7bec327afaff68953 (table rebalancer_dynamic_table_new_name [id=e730f17f1ded41b79e871afc5cacb78a]) successfully deleted
I20240501 08:12:10.127460 27613 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_extra_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:10.135963 30916 tablet_service.cc:1467] Processing CreateTablet for tablet 948f6afb45594870bfe44b45d303f88f (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=7c8060b6145d492cadb48df007ce38c9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:10.136467 30916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 948f6afb45594870bfe44b45d303f88f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:10.136857 28132 tablet_service.cc:1467] Processing CreateTablet for tablet b6ad187384514267b57cc994c9612709 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=7c8060b6145d492cadb48df007ce38c9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:10.136893 30517 tablet_service.cc:1467] Processing CreateTablet for tablet 948f6afb45594870bfe44b45d303f88f (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=7c8060b6145d492cadb48df007ce38c9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:10.137279 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 948f6afb45594870bfe44b45d303f88f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:10.137288 28132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6ad187384514267b57cc994c9612709. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:10.138140 30915 tablet_service.cc:1467] Processing CreateTablet for tablet b5d7223054aa4308917fd402d65ded5d (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=7c8060b6145d492cadb48df007ce38c9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:10.138551 30915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5d7223054aa4308917fd402d65ded5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:10.142410 30917 tablet_service.cc:1467] Processing CreateTablet for tablet b6ad187384514267b57cc994c9612709 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=7c8060b6145d492cadb48df007ce38c9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:10.142868 30917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6ad187384514267b57cc994c9612709. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:10.143064 30652 tablet_service.cc:1467] Processing CreateTablet for tablet b5d7223054aa4308917fd402d65ded5d (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=7c8060b6145d492cadb48df007ce38c9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:10.143486 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5d7223054aa4308917fd402d65ded5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:10.146739 30784 tablet_service.cc:1467] Processing CreateTablet for tablet b6ad187384514267b57cc994c9612709 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=7c8060b6145d492cadb48df007ce38c9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:10.147162 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6ad187384514267b57cc994c9612709. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:10.148116 27871 tablet_service.cc:1467] Processing CreateTablet for tablet 948f6afb45594870bfe44b45d303f88f (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=7c8060b6145d492cadb48df007ce38c9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:10.148610 27871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 948f6afb45594870bfe44b45d303f88f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:10.152849 30514 tablet_service.cc:1467] Processing CreateTablet for tablet b5d7223054aa4308917fd402d65ded5d (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=7c8060b6145d492cadb48df007ce38c9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:10.153355 30514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5d7223054aa4308917fd402d65ded5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:10.153767 31348 consensus_queue.cc:1035] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:10.166682 31348 consensus_queue.cc:1035] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:10.170217 31484 tablet_bootstrap.cc:492] T 948f6afb45594870bfe44b45d303f88f P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:10.170816 31481 tablet_bootstrap.cc:492] T 948f6afb45594870bfe44b45d303f88f P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:10.172873 31485 tablet_bootstrap.cc:492] T 948f6afb45594870bfe44b45d303f88f P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:10.172919 31484 tablet_bootstrap.cc:654] T 948f6afb45594870bfe44b45d303f88f P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:10.173069 31481 tablet_bootstrap.cc:654] T 948f6afb45594870bfe44b45d303f88f P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:10.176319 31485 tablet_bootstrap.cc:654] T 948f6afb45594870bfe44b45d303f88f P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:10.181591 31485 tablet_bootstrap.cc:492] T 948f6afb45594870bfe44b45d303f88f P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:12:10.182343 31485 ts_tablet_manager.cc:1367] T 948f6afb45594870bfe44b45d303f88f P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240501 08:12:10.183486 31485 raft_consensus.cc:348] T 948f6afb45594870bfe44b45d303f88f P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:10.185091 31485 raft_consensus.cc:374] T 948f6afb45594870bfe44b45d303f88f P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:10.185448 31485 raft_consensus.cc:729] T 948f6afb45594870bfe44b45d303f88f P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:12:10.186115 31485 consensus_queue.cc:260] T 948f6afb45594870bfe44b45d303f88f P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:10.187429 31485 ts_tablet_manager.cc:1398] T 948f6afb45594870bfe44b45d303f88f P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:12:10.189075 31484 tablet_bootstrap.cc:492] T 948f6afb45594870bfe44b45d303f88f P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:10.190142 31484 ts_tablet_manager.cc:1367] T 948f6afb45594870bfe44b45d303f88f P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.005s
I20240501 08:12:10.191316 31484 raft_consensus.cc:348] T 948f6afb45594870bfe44b45d303f88f P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:10.192873 31484 raft_consensus.cc:374] T 948f6afb45594870bfe44b45d303f88f P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:10.193168 31484 raft_consensus.cc:729] T 948f6afb45594870bfe44b45d303f88f P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:10.193646 31481 tablet_bootstrap.cc:492] T 948f6afb45594870bfe44b45d303f88f P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:10.193842 31481 ts_tablet_manager.cc:1367] T 948f6afb45594870bfe44b45d303f88f P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.014s
I20240501 08:12:10.193634 31484 consensus_queue.cc:260] T 948f6afb45594870bfe44b45d303f88f P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:10.195276 31481 raft_consensus.cc:348] T 948f6afb45594870bfe44b45d303f88f P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:10.195511 31481 raft_consensus.cc:374] T 948f6afb45594870bfe44b45d303f88f P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:10.195613 31481 raft_consensus.cc:729] T 948f6afb45594870bfe44b45d303f88f P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:10.195865 31481 consensus_queue.cc:260] T 948f6afb45594870bfe44b45d303f88f P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:10.199046 31481 ts_tablet_manager.cc:1398] T 948f6afb45594870bfe44b45d303f88f P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 08:12:10.200274 31481 tablet_bootstrap.cc:492] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:10.204404 31481 tablet_bootstrap.cc:654] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:10.204406 31484 ts_tablet_manager.cc:1398] T 948f6afb45594870bfe44b45d303f88f P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.014s user 0.003s sys 0.002s
I20240501 08:12:10.207322 31484 tablet_bootstrap.cc:492] T b5d7223054aa4308917fd402d65ded5d P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:10.211618 31484 tablet_bootstrap.cc:654] T b5d7223054aa4308917fd402d65ded5d P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:10.214864 31481 tablet_bootstrap.cc:492] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:10.215168 31481 ts_tablet_manager.cc:1367] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.007s
I20240501 08:12:10.216208 31481 raft_consensus.cc:348] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:10.216537 31481 raft_consensus.cc:374] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:10.216668 31481 raft_consensus.cc:729] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:10.217022 31481 consensus_queue.cc:260] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:10.217953 27613 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:10.223110 31492 tablet_bootstrap.cc:492] T b6ad187384514267b57cc994c9612709 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:10.224840 31484 tablet_bootstrap.cc:492] T b5d7223054aa4308917fd402d65ded5d P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:10.226152 31484 ts_tablet_manager.cc:1367] T b5d7223054aa4308917fd402d65ded5d P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.001s
I20240501 08:12:10.227478 31484 raft_consensus.cc:348] T b5d7223054aa4308917fd402d65ded5d P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:10.227831 31492 tablet_bootstrap.cc:654] T b6ad187384514267b57cc994c9612709 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:10.228500 31457 raft_consensus.cc:482] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:10.228667 31457 raft_consensus.cc:504] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:10.229187 31484 raft_consensus.cc:374] T b5d7223054aa4308917fd402d65ded5d P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:10.229300 31484 raft_consensus.cc:729] T b5d7223054aa4308917fd402d65ded5d P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:10.229403 31457 leader_election.cc:288] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:10.229585 31484 consensus_queue.cc:260] T b5d7223054aa4308917fd402d65ded5d P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:10.230551 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5d7223054aa4308917fd402d65ded5d" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
W20240501 08:12:10.231210 30870 leader_election.cc:341] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:10.231333 31484 ts_tablet_manager.cc:1398] T b5d7223054aa4308917fd402d65ded5d P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:12:10.231757 31493 tablet_bootstrap.cc:492] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:10.234545 31493 tablet_bootstrap.cc:654] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:10.235733 31481 ts_tablet_manager.cc:1398] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20240501 08:12:10.238343 31481 tablet_bootstrap.cc:492] T b6ad187384514267b57cc994c9612709 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:10.241459 30517 tablet_service.cc:1467] Processing CreateTablet for tablet 559f6f3bd66945b884bb3b4c3c57c97f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8297f2455ec2481fb612dcead6459323]), partition=
I20240501 08:12:10.241575 31481 tablet_bootstrap.cc:654] T b6ad187384514267b57cc994c9612709 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:10.241844 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 559f6f3bd66945b884bb3b4c3c57c97f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:10.248466 31481 tablet_bootstrap.cc:492] T b6ad187384514267b57cc994c9612709 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:10.248713 31481 ts_tablet_manager.cc:1367] T b6ad187384514267b57cc994c9612709 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20240501 08:12:10.249867 30651 tablet_service.cc:1467] Processing CreateTablet for tablet 559f6f3bd66945b884bb3b4c3c57c97f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8297f2455ec2481fb612dcead6459323]), partition=
I20240501 08:12:10.250090 31481 raft_consensus.cc:348] T b6ad187384514267b57cc994c9612709 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:10.250288 30651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 559f6f3bd66945b884bb3b4c3c57c97f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:10.250303 31481 raft_consensus.cc:374] T b6ad187384514267b57cc994c9612709 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:10.250387 31481 raft_consensus.cc:729] T b6ad187384514267b57cc994c9612709 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:10.250639 31481 consensus_queue.cc:260] T b6ad187384514267b57cc994c9612709 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:10.251344 31481 ts_tablet_manager.cc:1398] T b6ad187384514267b57cc994c9612709 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:10.271683 31484 tablet_bootstrap.cc:492] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:10.273885 31484 tablet_bootstrap.cc:654] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:10.279680 31484 tablet_bootstrap.cc:492] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:10.279932 31484 ts_tablet_manager.cc:1367] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20240501 08:12:10.281180 31484 raft_consensus.cc:348] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:10.281493 31484 raft_consensus.cc:374] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:10.281641 31484 raft_consensus.cc:729] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:10.281925 31484 consensus_queue.cc:260] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:10.282738 31484 ts_tablet_manager.cc:1398] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:10.274605 28132 tablet_service.cc:1467] Processing CreateTablet for tablet 559f6f3bd66945b884bb3b4c3c57c97f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8297f2455ec2481fb612dcead6459323]), partition=
I20240501 08:12:10.284852 28132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 559f6f3bd66945b884bb3b4c3c57c97f. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 08:12:10.287276 30472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.136:46845, user_credentials={real_user=slave}} blocked reactor thread for 112668us
I20240501 08:12:10.288614 31493 tablet_bootstrap.cc:492] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:10.288503 30535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5d7223054aa4308917fd402d65ded5d" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:10.288766 30535 raft_consensus.cc:2454] T b5d7223054aa4308917fd402d65ded5d P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:10.288995 31493 ts_tablet_manager.cc:1367] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.061s user 0.004s sys 0.000s
I20240501 08:12:10.289846 31493 raft_consensus.cc:348] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:10.291358 31492 tablet_bootstrap.cc:492] T b6ad187384514267b57cc994c9612709 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:12:10.291570 31492 ts_tablet_manager.cc:1367] T b6ad187384514267b57cc994c9612709 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.069s user 0.005s sys 0.000s
I20240501 08:12:10.291887 30873 leader_election.cc:302] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters: 9a022e1a501f4d44a0468054d6c3de47
I20240501 08:12:10.292196 31493 raft_consensus.cc:374] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:10.292340 31493 raft_consensus.cc:729] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:10.292403 31457 raft_consensus.cc:2781] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:10.292528 31457 raft_consensus.cc:482] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:10.292603 31457 raft_consensus.cc:3037] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:10.292863 31493 consensus_queue.cc:260] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:10.293562 31492 raft_consensus.cc:348] T b6ad187384514267b57cc994c9612709 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:10.295760 31457 raft_consensus.cc:504] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:10.297911 31493 ts_tablet_manager.cc:1398] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240501 08:12:10.298116 31492 raft_consensus.cc:374] T b6ad187384514267b57cc994c9612709 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:10.298240 31492 raft_consensus.cc:729] T b6ad187384514267b57cc994c9612709 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:12:10.298499 31492 consensus_queue.cc:260] T b6ad187384514267b57cc994c9612709 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:10.300308 31457 leader_election.cc:288] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Requested vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:10.301419 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5d7223054aa4308917fd402d65ded5d" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:10.301411 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5d7223054aa4308917fd402d65ded5d" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:10.301617 30537 raft_consensus.cc:3037] T b5d7223054aa4308917fd402d65ded5d P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
W20240501 08:12:10.302181 30870 leader_election.cc:341] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:10.302651 31482 tablet_bootstrap.cc:492] T b5d7223054aa4308917fd402d65ded5d P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:10.303723 30537 raft_consensus.cc:2454] T b5d7223054aa4308917fd402d65ded5d P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:10.304148 30873 leader_election.cc:302] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters: 9a022e1a501f4d44a0468054d6c3de47
I20240501 08:12:10.305145 31482 tablet_bootstrap.cc:654] T b5d7223054aa4308917fd402d65ded5d P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:10.306300 31457 raft_consensus.cc:2781] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:10.306667 31492 ts_tablet_manager.cc:1398] T b6ad187384514267b57cc994c9612709 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240501 08:12:10.334380 31492 tablet_bootstrap.cc:492] T 559f6f3bd66945b884bb3b4c3c57c97f P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:10.336741 31492 tablet_bootstrap.cc:654] T 559f6f3bd66945b884bb3b4c3c57c97f P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:10.342162 31457 raft_consensus.cc:686] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Becoming Leader. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:12:10.342651 31457 consensus_queue.cc:237] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
W20240501 08:12:10.348492 27957 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.129:43471, user_credentials={real_user=slave}} blocked reactor thread for 47562.2us
I20240501 08:12:10.349146 31482 tablet_bootstrap.cc:492] T b5d7223054aa4308917fd402d65ded5d P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:10.349292 31492 tablet_bootstrap.cc:492] T 559f6f3bd66945b884bb3b4c3c57c97f P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:12:10.349414 31482 ts_tablet_manager.cc:1367] T b5d7223054aa4308917fd402d65ded5d P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.047s user 0.008s sys 0.033s
I20240501 08:12:10.349467 31492 ts_tablet_manager.cc:1367] T 559f6f3bd66945b884bb3b4c3c57c97f P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.006s
I20240501 08:12:10.350610 31492 raft_consensus.cc:348] T 559f6f3bd66945b884bb3b4c3c57c97f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:10.350809 31492 raft_consensus.cc:374] T 559f6f3bd66945b884bb3b4c3c57c97f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:10.350929 31492 raft_consensus.cc:729] T 559f6f3bd66945b884bb3b4c3c57c97f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:12:10.351171 31492 consensus_queue.cc:260] T 559f6f3bd66945b884bb3b4c3c57c97f P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:10.351785 31492 ts_tablet_manager.cc:1398] T 559f6f3bd66945b884bb3b4c3c57c97f P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:10.352370 31482 raft_consensus.cc:348] T b5d7223054aa4308917fd402d65ded5d P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:10.352617 31482 raft_consensus.cc:374] T b5d7223054aa4308917fd402d65ded5d P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:10.352833 31482 raft_consensus.cc:729] T b5d7223054aa4308917fd402d65ded5d P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:10.353099 31482 consensus_queue.cc:260] T b5d7223054aa4308917fd402d65ded5d P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:10.353303 27617 catalog_manager.cc:5484] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 reported cstate change: term changed from 0 to 1, leader changed from <none> to a29cd590999b4f32849ff5689cd53c11 (127.21.84.136). New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:10.353720 31482 ts_tablet_manager.cc:1398] T b5d7223054aa4308917fd402d65ded5d P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:12:10.354568 31482 tablet_bootstrap.cc:492] T 559f6f3bd66945b884bb3b4c3c57c97f P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:10.356009 31482 tablet_bootstrap.cc:654] T 559f6f3bd66945b884bb3b4c3c57c97f P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:10.357873 31482 tablet_bootstrap.cc:492] T 559f6f3bd66945b884bb3b4c3c57c97f P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:10.358855 31482 ts_tablet_manager.cc:1367] T 559f6f3bd66945b884bb3b4c3c57c97f P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240501 08:12:10.360081 31482 raft_consensus.cc:348] T 559f6f3bd66945b884bb3b4c3c57c97f P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:10.360298 31482 raft_consensus.cc:374] T 559f6f3bd66945b884bb3b4c3c57c97f P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:10.360381 31482 raft_consensus.cc:729] T 559f6f3bd66945b884bb3b4c3c57c97f P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:10.360579 31482 consensus_queue.cc:260] T 559f6f3bd66945b884bb3b4c3c57c97f P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:10.361155 31482 ts_tablet_manager.cc:1398] T 559f6f3bd66945b884bb3b4c3c57c97f P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240501 08:12:10.406159 31351 raft_consensus.cc:482] T b5d7223054aa4308917fd402d65ded5d P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:10.406368 31351 raft_consensus.cc:504] T b5d7223054aa4308917fd402d65ded5d P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:10.407119 31351 leader_election.cc:288] T b5d7223054aa4308917fd402d65ded5d P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:10.407577 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5d7223054aa4308917fd402d65ded5d" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:10.407825 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5d7223054aa4308917fd402d65ded5d" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:10.408123 30537 raft_consensus.cc:2379] T b5d7223054aa4308917fd402d65ded5d P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a022e1a501f4d44a0468054d6c3de47 in current term 1: Already voted for candidate a29cd590999b4f32849ff5689cd53c11 in this term.
I20240501 08:12:10.408504 30608 leader_election.cc:302] T b5d7223054aa4308917fd402d65ded5d P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a022e1a501f4d44a0468054d6c3de47; no voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11
I20240501 08:12:10.408792 31351 raft_consensus.cc:3037] T b5d7223054aa4308917fd402d65ded5d P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:10.410792 31351 raft_consensus.cc:2726] T b5d7223054aa4308917fd402d65ded5d P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:10.417457 31497 raft_consensus.cc:482] T 948f6afb45594870bfe44b45d303f88f P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:10.417627 31497 raft_consensus.cc:504] T 948f6afb45594870bfe44b45d303f88f P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:10.418330 31497 leader_election.cc:288] T 948f6afb45594870bfe44b45d303f88f P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:10.418828 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "948f6afb45594870bfe44b45d303f88f" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:10.418792 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "948f6afb45594870bfe44b45d303f88f" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:10.419041 30936 raft_consensus.cc:2454] T 948f6afb45594870bfe44b45d303f88f P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:12:10.419062 30537 raft_consensus.cc:2454] T 948f6afb45594870bfe44b45d303f88f P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:12:10.419430 27828 leader_election.cc:302] T 948f6afb45594870bfe44b45d303f88f P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:10.419664 31497 raft_consensus.cc:2781] T 948f6afb45594870bfe44b45d303f88f P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:10.419776 31497 raft_consensus.cc:482] T 948f6afb45594870bfe44b45d303f88f P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:10.419847 31497 raft_consensus.cc:3037] T 948f6afb45594870bfe44b45d303f88f P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:10.422470 31497 raft_consensus.cc:504] T 948f6afb45594870bfe44b45d303f88f P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:10.423063 31497 leader_election.cc:288] T 948f6afb45594870bfe44b45d303f88f P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Requested vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:10.423530 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "948f6afb45594870bfe44b45d303f88f" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:10.423594 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "948f6afb45594870bfe44b45d303f88f" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:10.423781 30537 raft_consensus.cc:3037] T 948f6afb45594870bfe44b45d303f88f P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:10.423797 30936 raft_consensus.cc:3037] T 948f6afb45594870bfe44b45d303f88f P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:10.426653 30936 raft_consensus.cc:2454] T 948f6afb45594870bfe44b45d303f88f P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:12:10.426661 30537 raft_consensus.cc:2454] T 948f6afb45594870bfe44b45d303f88f P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:12:10.427110 27828 leader_election.cc:302] T 948f6afb45594870bfe44b45d303f88f P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:10.427395 31497 raft_consensus.cc:2781] T 948f6afb45594870bfe44b45d303f88f P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:10.427522 31497 raft_consensus.cc:686] T 948f6afb45594870bfe44b45d303f88f P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Becoming Leader. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Running, Role: LEADER
I20240501 08:12:10.427840 31497 consensus_queue.cc:237] T 948f6afb45594870bfe44b45d303f88f P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:10.430867 27613 catalog_manager.cc:5484] T 948f6afb45594870bfe44b45d303f88f P f1525cbc82a24d13b65f136695c903af reported cstate change: term changed from 0 to 1, leader changed from <none> to f1525cbc82a24d13b65f136695c903af (127.21.84.130). New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:10.489184 31348 raft_consensus.cc:482] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:10.489392 31348 raft_consensus.cc:504] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:10.490101 31348 leader_election.cc:288] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:10.490638 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559f6f3bd66945b884bb3b4c3c57c97f" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:10.490964 30672 raft_consensus.cc:2454] T 559f6f3bd66945b884bb3b4c3c57c97f P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 0.
I20240501 08:12:10.491478 30470 leader_election.cc:302] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:10.491786 31348 raft_consensus.cc:2781] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:10.491878 31348 raft_consensus.cc:482] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:10.491993 31348 raft_consensus.cc:3037] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:10.495180 31348 raft_consensus.cc:504] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:10.496613 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559f6f3bd66945b884bb3b4c3c57c97f" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:10.496742 31348 leader_election.cc:288] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 election: Requested vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:10.496850 30672 raft_consensus.cc:3037] T 559f6f3bd66945b884bb3b4c3c57c97f P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:10.499058 30672 raft_consensus.cc:2454] T 559f6f3bd66945b884bb3b4c3c57c97f P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 1.
I20240501 08:12:10.499014 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559f6f3bd66945b884bb3b4c3c57c97f" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:12:10.499083 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559f6f3bd66945b884bb3b4c3c57c97f" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:12:10.499279 28151 raft_consensus.cc:2454] T 559f6f3bd66945b884bb3b4c3c57c97f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 0.
I20240501 08:12:10.499500 30470 leader_election.cc:302] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:10.499791 31348 raft_consensus.cc:2781] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:10.499989 31348 raft_consensus.cc:686] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Becoming Leader. State: Replica: 31811708c6774037b82df44024b229b6, State: Running, Role: LEADER
I20240501 08:12:10.500311 31348 consensus_queue.cc:237] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:10.502664 31476 raft_consensus.cc:482] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:10.502835 31476 raft_consensus.cc:504] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:10.503542 31476 leader_election.cc:288] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:10.503993 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ad187384514267b57cc994c9612709" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:12:10.504215 28151 raft_consensus.cc:2454] T b6ad187384514267b57cc994c9612709 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:10.504206 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ad187384514267b57cc994c9612709" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:10.504396 30936 raft_consensus.cc:2454] T b6ad187384514267b57cc994c9612709 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:10.504637 30737 leader_election.cc:302] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:12:10.504932 31476 raft_consensus.cc:2781] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:10.505016 31476 raft_consensus.cc:482] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:10.505123 31476 raft_consensus.cc:3037] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:10.505084 27613 catalog_manager.cc:5484] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31811708c6774037b82df44024b229b6 (127.21.84.133). New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:10.507495 31476 raft_consensus.cc:504] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:10.508723 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ad187384514267b57cc994c9612709" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:12:10.508972 28151 raft_consensus.cc:3037] T b6ad187384514267b57cc994c9612709 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:10.508944 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ad187384514267b57cc994c9612709" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:10.509138 30936 raft_consensus.cc:3037] T b6ad187384514267b57cc994c9612709 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:10.511674 28151 raft_consensus.cc:2454] T b6ad187384514267b57cc994c9612709 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:10.511688 30936 raft_consensus.cc:2454] T b6ad187384514267b57cc994c9612709 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:10.512059 30737 leader_election.cc:302] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:12:10.508234 31476 leader_election.cc:288] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Requested vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:10.512375 31458 raft_consensus.cc:2781] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:10.512573 31458 raft_consensus.cc:686] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Becoming Leader. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Running, Role: LEADER
I20240501 08:12:10.512818 31458 consensus_queue.cc:237] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:10.515635 27613 catalog_manager.cc:5484] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64998323a1f4482d8704bf20594d13a5 (127.21.84.135). New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:10.639779 27613 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:10.645164 28151 raft_consensus.cc:3037] T 559f6f3bd66945b884bb3b4c3c57c97f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:10.645367 30672 raft_consensus.cc:1261] T 559f6f3bd66945b884bb3b4c3c57c97f P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:10.645956 31348 consensus_queue.cc:1035] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:10.647864 28151 raft_consensus.cc:1261] T 559f6f3bd66945b884bb3b4c3c57c97f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:10.649186 31487 consensus_queue.cc:1035] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:10.649713 27613 catalog_manager.cc:5576] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933) does not have the latest schema for tablet 559f6f3bd66945b884bb3b4c3c57c97f (table rebalancer_dynamic_table [id=8297f2455ec2481fb612dcead6459323]). Expected version 1 got 0
I20240501 08:12:10.651786 31517 tablet.cc:1722] T 559f6f3bd66945b884bb3b4c3c57c97f P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:10.652449 31517 tablet.cc:1620] T 559f6f3bd66945b884bb3b4c3c57c97f P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:10.652675 31516 tablet.cc:1722] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:10.653221 31516 tablet.cc:1620] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:10.653431 27617 catalog_manager.cc:5576] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933) does not have the latest schema for tablet 559f6f3bd66945b884bb3b4c3c57c97f (table rebalancer_dynamic_table [id=8297f2455ec2481fb612dcead6459323]). Expected version 1 got 0
I20240501 08:12:10.661216 27617 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=8297f2455ec2481fb612dcead6459323] alter complete (version 1)
I20240501 08:12:10.666090 31520 tablet.cc:1722] T 559f6f3bd66945b884bb3b4c3c57c97f P 91d11eae5ef6432ca86eb8ed446b6ea6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:10.667271 31520 tablet.cc:1620] T 559f6f3bd66945b884bb3b4c3c57c97f P 91d11eae5ef6432ca86eb8ed446b6ea6: MemRowSet was empty: no flush needed.
I20240501 08:12:10.668311 27620 catalog_manager.cc:5576] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323) does not have the latest schema for tablet 559f6f3bd66945b884bb3b4c3c57c97f (table rebalancer_dynamic_table [id=8297f2455ec2481fb612dcead6459323]). Expected version 1 got 0
I20240501 08:12:10.766803 27620 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:10.773342 27620 catalog_manager.cc:5576] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293) does not have the latest schema for tablet 559f6f3bd66945b884bb3b4c3c57c97f (table rebalancer_dynamic_table [id=8297f2455ec2481fb612dcead6459323]). Expected version 2 got 1
I20240501 08:12:10.773986 31516 tablet.cc:1722] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:10.774529 31516 tablet.cc:1620] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:10.774551 31517 tablet.cc:1722] T 559f6f3bd66945b884bb3b4c3c57c97f P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:10.774978 31517 tablet.cc:1620] T 559f6f3bd66945b884bb3b4c3c57c97f P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:10.776453 31520 tablet.cc:1722] T 559f6f3bd66945b884bb3b4c3c57c97f P 91d11eae5ef6432ca86eb8ed446b6ea6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:10.776918 31520 tablet.cc:1620] T 559f6f3bd66945b884bb3b4c3c57c97f P 91d11eae5ef6432ca86eb8ed446b6ea6: MemRowSet was empty: no flush needed.
I20240501 08:12:10.783703 27617 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=8297f2455ec2481fb612dcead6459323] alter complete (version 2)
I20240501 08:12:10.795838 31457 consensus_queue.cc:1035] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:10.801120 31523 consensus_queue.cc:1035] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:10.854074 31376 consensus_queue.cc:1035] T 948f6afb45594870bfe44b45d303f88f P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:10.858703 31376 consensus_queue.cc:1035] T 948f6afb45594870bfe44b45d303f88f P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:10.886042 27617 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:10.894028 31516 tablet.cc:1722] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:10.894336 31517 tablet.cc:1722] T 559f6f3bd66945b884bb3b4c3c57c97f P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:10.894388 31520 tablet.cc:1722] T 559f6f3bd66945b884bb3b4c3c57c97f P 91d11eae5ef6432ca86eb8ed446b6ea6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:10.902482 27617 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table_new_name [id=8297f2455ec2481fb612dcead6459323] alter complete (version 3)
I20240501 08:12:10.905203 31458 consensus_queue.cc:1035] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:10.915941 31458 consensus_queue.cc:1035] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:11.006352 27617 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:11.006568 27617 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:11.009683 27617 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 559f6f3bd66945b884bb3b4c3c57c97f
I20240501 08:12:11.010540 30652 tablet_service.cc:1514] Processing DeleteTablet for tablet 559f6f3bd66945b884bb3b4c3c57c97f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:11 UTC) from {username='slave'} at 127.0.0.1:57156
I20240501 08:12:11.011117 28132 tablet_service.cc:1514] Processing DeleteTablet for tablet 559f6f3bd66945b884bb3b4c3c57c97f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:11 UTC) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:11.011317 30517 tablet_service.cc:1514] Processing DeleteTablet for tablet 559f6f3bd66945b884bb3b4c3c57c97f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:11 UTC) from {username='slave'} at 127.0.0.1:45078
I20240501 08:12:11.011766 31533 tablet_replica.cc:330] T 559f6f3bd66945b884bb3b4c3c57c97f P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:11.012045 31532 tablet_replica.cc:330] T 559f6f3bd66945b884bb3b4c3c57c97f P 9a022e1a501f4d44a0468054d6c3de47: stopping tablet replica
I20240501 08:12:11.012070 31533 raft_consensus.cc:2229] T 559f6f3bd66945b884bb3b4c3c57c97f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:11.012156 31534 tablet_replica.cc:330] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6: stopping tablet replica
I20240501 08:12:11.012372 31534 raft_consensus.cc:2229] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:11.012391 31533 raft_consensus.cc:2258] T 559f6f3bd66945b884bb3b4c3c57c97f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:11.012398 31532 raft_consensus.cc:2229] T 559f6f3bd66945b884bb3b4c3c57c97f P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:11.012907 31534 raft_consensus.cc:2258] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:11.013115 31532 raft_consensus.cc:2258] T 559f6f3bd66945b884bb3b4c3c57c97f P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:11.013291 31533 ts_tablet_manager.cc:1875] T 559f6f3bd66945b884bb3b4c3c57c97f P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:11.013689 31534 ts_tablet_manager.cc:1875] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:11.014122 31532 ts_tablet_manager.cc:1875] T 559f6f3bd66945b884bb3b4c3c57c97f P 9a022e1a501f4d44a0468054d6c3de47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:11.018026 31533 ts_tablet_manager.cc:1888] T 559f6f3bd66945b884bb3b4c3c57c97f P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:11.018167 31533 log.cc:1196] T 559f6f3bd66945b884bb3b4c3c57c97f P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/559f6f3bd66945b884bb3b4c3c57c97f
I20240501 08:12:11.018543 31533 ts_tablet_manager.cc:1909] T 559f6f3bd66945b884bb3b4c3c57c97f P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting consensus metadata
I20240501 08:12:11.019541 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet 559f6f3bd66945b884bb3b4c3c57c97f (table rebalancer_dynamic_table_new_name [id=8297f2455ec2481fb612dcead6459323]) successfully deleted
I20240501 08:12:11.020197 31534 ts_tablet_manager.cc:1888] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:11.020336 31534 log.cc:1196] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/wals/559f6f3bd66945b884bb3b4c3c57c97f
I20240501 08:12:11.020371 31532 ts_tablet_manager.cc:1888] T 559f6f3bd66945b884bb3b4c3c57c97f P 9a022e1a501f4d44a0468054d6c3de47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:11.020519 31532 log.cc:1196] T 559f6f3bd66945b884bb3b4c3c57c97f P 9a022e1a501f4d44a0468054d6c3de47: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/wals/559f6f3bd66945b884bb3b4c3c57c97f
I20240501 08:12:11.020825 31534 ts_tablet_manager.cc:1909] T 559f6f3bd66945b884bb3b4c3c57c97f P 31811708c6774037b82df44024b229b6: Deleting consensus metadata
I20240501 08:12:11.020972 31532 ts_tablet_manager.cc:1909] T 559f6f3bd66945b884bb3b4c3c57c97f P 9a022e1a501f4d44a0468054d6c3de47: Deleting consensus metadata
I20240501 08:12:11.021984 27607 catalog_manager.cc:4830] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): tablet 559f6f3bd66945b884bb3b4c3c57c97f (table rebalancer_dynamic_table_new_name [id=8297f2455ec2481fb612dcead6459323]) successfully deleted
I20240501 08:12:11.022054 27604 catalog_manager.cc:4830] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): tablet 559f6f3bd66945b884bb3b4c3c57c97f (table rebalancer_dynamic_table_new_name [id=8297f2455ec2481fb612dcead6459323]) successfully deleted
I20240501 08:12:11.088651 27617 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_extra_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:11.097716 27871 tablet_service.cc:1467] Processing CreateTablet for tablet 6a5f99276dad456ea80e3c9f9618059a (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=bd2ed215ec044ce782fe6f7c802797e6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:11.098239 27871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a5f99276dad456ea80e3c9f9618059a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.098238 30784 tablet_service.cc:1467] Processing CreateTablet for tablet 6a5f99276dad456ea80e3c9f9618059a (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=bd2ed215ec044ce782fe6f7c802797e6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:11.098364 30781 tablet_service.cc:1467] Processing CreateTablet for tablet 8277d70d49f54197af60e0be86d34a9b (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=bd2ed215ec044ce782fe6f7c802797e6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:11.098672 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a5f99276dad456ea80e3c9f9618059a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.099323 30782 tablet_service.cc:1467] Processing CreateTablet for tablet c4851258508d46c88493503a6257f365 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=bd2ed215ec044ce782fe6f7c802797e6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:11.099726 30782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4851258508d46c88493503a6257f365. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.102862 27739 tablet_service.cc:1467] Processing CreateTablet for tablet 6a5f99276dad456ea80e3c9f9618059a (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=bd2ed215ec044ce782fe6f7c802797e6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:11.103300 27739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a5f99276dad456ea80e3c9f9618059a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.103667 30517 tablet_service.cc:1467] Processing CreateTablet for tablet 8277d70d49f54197af60e0be86d34a9b (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=bd2ed215ec044ce782fe6f7c802797e6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:11.103996 30917 tablet_service.cc:1467] Processing CreateTablet for tablet 8277d70d49f54197af60e0be86d34a9b (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=bd2ed215ec044ce782fe6f7c802797e6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:11.104117 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8277d70d49f54197af60e0be86d34a9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.104387 30917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8277d70d49f54197af60e0be86d34a9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.106130 30781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8277d70d49f54197af60e0be86d34a9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.108116 30916 tablet_service.cc:1467] Processing CreateTablet for tablet c4851258508d46c88493503a6257f365 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=bd2ed215ec044ce782fe6f7c802797e6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:11.108533 30916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4851258508d46c88493503a6257f365. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.112082 27613 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:11.122440 27872 tablet_service.cc:1467] Processing CreateTablet for tablet c4851258508d46c88493503a6257f365 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=bd2ed215ec044ce782fe6f7c802797e6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:11.122967 27872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4851258508d46c88493503a6257f365. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.132853 30915 tablet_service.cc:1467] Processing CreateTablet for tablet c89badd2c37f4f6e888b86b0e4226bd8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=315b3679582b4b608aff3d4852535612]), partition=
I20240501 08:12:11.133369 30915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c89badd2c37f4f6e888b86b0e4226bd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.133607 30782 tablet_service.cc:1467] Processing CreateTablet for tablet c89badd2c37f4f6e888b86b0e4226bd8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=315b3679582b4b608aff3d4852535612]), partition=
I20240501 08:12:11.134068 30782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c89badd2c37f4f6e888b86b0e4226bd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.138908 27741 tablet_service.cc:1467] Processing CreateTablet for tablet c89badd2c37f4f6e888b86b0e4226bd8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=315b3679582b4b608aff3d4852535612]), partition=
I20240501 08:12:11.139393 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c89badd2c37f4f6e888b86b0e4226bd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.141048 31540 tablet_bootstrap.cc:492] T 8277d70d49f54197af60e0be86d34a9b P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:11.143882 31537 tablet_bootstrap.cc:492] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:11.144443 31539 tablet_bootstrap.cc:492] T 6a5f99276dad456ea80e3c9f9618059a P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:11.147302 31537 tablet_bootstrap.cc:654] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:11.149358 31539 tablet_bootstrap.cc:654] T 6a5f99276dad456ea80e3c9f9618059a P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:11.149472 31540 tablet_bootstrap.cc:654] T 8277d70d49f54197af60e0be86d34a9b P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:11.156273 31539 tablet_bootstrap.cc:492] T 6a5f99276dad456ea80e3c9f9618059a P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:12:11.157708 31539 ts_tablet_manager.cc:1367] T 6a5f99276dad456ea80e3c9f9618059a P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240501 08:12:11.158879 31539 raft_consensus.cc:348] T 6a5f99276dad456ea80e3c9f9618059a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:11.160106 31540 tablet_bootstrap.cc:492] T 8277d70d49f54197af60e0be86d34a9b P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:11.160300 31540 ts_tablet_manager.cc:1367] T 8277d70d49f54197af60e0be86d34a9b P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.004s
I20240501 08:12:11.160918 31539 raft_consensus.cc:374] T 6a5f99276dad456ea80e3c9f9618059a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:11.161088 31539 raft_consensus.cc:729] T 6a5f99276dad456ea80e3c9f9618059a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:12:11.162078 31539 consensus_queue.cc:260] T 6a5f99276dad456ea80e3c9f9618059a P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:11.162755 31539 ts_tablet_manager.cc:1398] T 6a5f99276dad456ea80e3c9f9618059a P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:12:11.161630 31540 raft_consensus.cc:348] T 8277d70d49f54197af60e0be86d34a9b P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:11.163050 31540 raft_consensus.cc:374] T 8277d70d49f54197af60e0be86d34a9b P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:11.163192 31540 raft_consensus.cc:729] T 8277d70d49f54197af60e0be86d34a9b P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:11.163487 31540 consensus_queue.cc:260] T 8277d70d49f54197af60e0be86d34a9b P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:11.163964 31539 tablet_bootstrap.cc:492] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:11.164532 31540 ts_tablet_manager.cc:1398] T 8277d70d49f54197af60e0be86d34a9b P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 08:12:11.167052 31540 tablet_bootstrap.cc:492] T c4851258508d46c88493503a6257f365 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:11.173064 31539 tablet_bootstrap.cc:654] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:11.169344 31540 tablet_bootstrap.cc:654] T c4851258508d46c88493503a6257f365 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:11.176594 31544 tablet_bootstrap.cc:492] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:11.180056 31539 tablet_bootstrap.cc:492] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:12:11.180343 31539 ts_tablet_manager.cc:1367] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240501 08:12:11.180433 31544 tablet_bootstrap.cc:654] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:11.181455 31539 raft_consensus.cc:348] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:11.181821 31539 raft_consensus.cc:374] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:11.181970 31539 raft_consensus.cc:729] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:12:11.182338 31539 consensus_queue.cc:260] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:11.183254 31539 ts_tablet_manager.cc:1398] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:11.185072 31537 tablet_bootstrap.cc:492] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:12:11.185227 31537 ts_tablet_manager.cc:1367] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.007s
I20240501 08:12:11.186450 31536 tablet_bootstrap.cc:492] T 8277d70d49f54197af60e0be86d34a9b P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:11.187296 31537 raft_consensus.cc:348] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:11.187564 31537 raft_consensus.cc:374] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:11.187673 31537 raft_consensus.cc:729] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:12:11.187920 31537 consensus_queue.cc:260] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:11.188501 31537 ts_tablet_manager.cc:1398] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 08:12:11.188947 31537 tablet_bootstrap.cc:492] T c89badd2c37f4f6e888b86b0e4226bd8 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:11.189301 31540 tablet_bootstrap.cc:492] T c4851258508d46c88493503a6257f365 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:11.189509 31540 ts_tablet_manager.cc:1367] T c4851258508d46c88493503a6257f365 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.004s
I20240501 08:12:11.190635 31537 tablet_bootstrap.cc:654] T c89badd2c37f4f6e888b86b0e4226bd8 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:11.190771 31536 tablet_bootstrap.cc:654] T 8277d70d49f54197af60e0be86d34a9b P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:11.190639 31540 raft_consensus.cc:348] T c4851258508d46c88493503a6257f365 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:11.190877 31540 raft_consensus.cc:374] T c4851258508d46c88493503a6257f365 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:11.190999 31540 raft_consensus.cc:729] T c4851258508d46c88493503a6257f365 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:11.191264 31540 consensus_queue.cc:260] T c4851258508d46c88493503a6257f365 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:11.191812 31540 ts_tablet_manager.cc:1398] T c4851258508d46c88493503a6257f365 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:12:11.192209 31540 tablet_bootstrap.cc:492] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:11.194453 31540 tablet_bootstrap.cc:654] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:11.194623 31458 raft_consensus.cc:482] T 8277d70d49f54197af60e0be86d34a9b P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:11.194761 31458 raft_consensus.cc:504] T 8277d70d49f54197af60e0be86d34a9b P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:11.195544 31458 leader_election.cc:288] T 8277d70d49f54197af60e0be86d34a9b P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:11.196378 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8277d70d49f54197af60e0be86d34a9b" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:11.197641 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8277d70d49f54197af60e0be86d34a9b" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
W20240501 08:12:11.198484 30740 leader_election.cc:341] T 8277d70d49f54197af60e0be86d34a9b P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:11.198707 31544 tablet_bootstrap.cc:492] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:11.198863 31544 ts_tablet_manager.cc:1367] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.005s
I20240501 08:12:11.199000 31535 raft_consensus.cc:482] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:11.199190 31535 raft_consensus.cc:504] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:11.200269 31535 leader_election.cc:288] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:11.200733 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5f99276dad456ea80e3c9f9618059a" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:12:11.201036 27889 raft_consensus.cc:2454] T 6a5f99276dad456ea80e3c9f9618059a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:12:11.201105 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5f99276dad456ea80e3c9f9618059a" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:11.201620 27697 leader_election.cc:302] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:11.202001 31535 raft_consensus.cc:2781] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:11.202144 31535 raft_consensus.cc:482] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:11.202240 31535 raft_consensus.cc:3037] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
W20240501 08:12:11.202807 30739 leader_election.cc:341] T 8277d70d49f54197af60e0be86d34a9b P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:11.202997 30739 leader_election.cc:302] T 8277d70d49f54197af60e0be86d34a9b P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5; no voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11
W20240501 08:12:11.204476 27695 leader_election.cc:341] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:11.208405 31458 raft_consensus.cc:2726] T 8277d70d49f54197af60e0be86d34a9b P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:11.210269 31535 raft_consensus.cc:504] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:11.211856 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5f99276dad456ea80e3c9f9618059a" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:12:11.212124 27889 raft_consensus.cc:3037] T 6a5f99276dad456ea80e3c9f9618059a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:11.196419 31536 tablet_bootstrap.cc:492] T 8277d70d49f54197af60e0be86d34a9b P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:11.215924 31536 ts_tablet_manager.cc:1367] T 8277d70d49f54197af60e0be86d34a9b P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20240501 08:12:11.217059 31536 raft_consensus.cc:348] T 8277d70d49f54197af60e0be86d34a9b P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:11.217352 31536 raft_consensus.cc:374] T 8277d70d49f54197af60e0be86d34a9b P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:11.217449 31536 raft_consensus.cc:729] T 8277d70d49f54197af60e0be86d34a9b P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:11.217767 31536 consensus_queue.cc:260] T 8277d70d49f54197af60e0be86d34a9b P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:11.220933 27889 raft_consensus.cc:2454] T 6a5f99276dad456ea80e3c9f9618059a P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:12:11.221627 27697 leader_election.cc:302] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:11.222597 31176 raft_consensus.cc:2781] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:11.223585 31536 ts_tablet_manager.cc:1398] T 8277d70d49f54197af60e0be86d34a9b P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240501 08:12:11.225548 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5f99276dad456ea80e3c9f9618059a" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
W20240501 08:12:11.226308 27695 leader_election.cc:341] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:11.229007 31535 leader_election.cc:288] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Requested vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:11.199852 31544 raft_consensus.cc:348] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:11.230500 31544 raft_consensus.cc:374] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:11.231359 31544 raft_consensus.cc:729] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:11.232128 31544 consensus_queue.cc:260] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:11.233614 31544 ts_tablet_manager.cc:1398] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.035s user 0.001s sys 0.001s
I20240501 08:12:11.238737 31540 tablet_bootstrap.cc:492] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:11.238935 31540 ts_tablet_manager.cc:1367] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.047s user 0.015s sys 0.022s
I20240501 08:12:11.239748 31540 raft_consensus.cc:348] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:11.239991 31540 raft_consensus.cc:374] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:11.240083 31540 raft_consensus.cc:729] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:11.240304 31540 consensus_queue.cc:260] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:11.240825 31540 ts_tablet_manager.cc:1398] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:11.241142 31540 tablet_bootstrap.cc:492] T 6a5f99276dad456ea80e3c9f9618059a P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:11.242733 31540 tablet_bootstrap.cc:654] T 6a5f99276dad456ea80e3c9f9618059a P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:11.251384 31537 tablet_bootstrap.cc:492] T c89badd2c37f4f6e888b86b0e4226bd8 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:12:11.251641 31537 ts_tablet_manager.cc:1367] T c89badd2c37f4f6e888b86b0e4226bd8 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.063s user 0.036s sys 0.020s
I20240501 08:12:11.252499 31537 raft_consensus.cc:348] T c89badd2c37f4f6e888b86b0e4226bd8 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:11.252789 31537 raft_consensus.cc:374] T c89badd2c37f4f6e888b86b0e4226bd8 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:11.252905 31537 raft_consensus.cc:729] T c89badd2c37f4f6e888b86b0e4226bd8 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:12:11.253185 31537 consensus_queue.cc:260] T c89badd2c37f4f6e888b86b0e4226bd8 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:11.253620 31540 tablet_bootstrap.cc:492] T 6a5f99276dad456ea80e3c9f9618059a P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:11.253849 31537 ts_tablet_manager.cc:1398] T c89badd2c37f4f6e888b86b0e4226bd8 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:11.254043 31176 raft_consensus.cc:686] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:12:11.254319 31176 consensus_queue.cc:237] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:11.254685 31540 ts_tablet_manager.cc:1367] T 6a5f99276dad456ea80e3c9f9618059a P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240501 08:12:11.255424 31540 raft_consensus.cc:348] T 6a5f99276dad456ea80e3c9f9618059a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:11.255621 31540 raft_consensus.cc:374] T 6a5f99276dad456ea80e3c9f9618059a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:11.255681 31540 raft_consensus.cc:729] T 6a5f99276dad456ea80e3c9f9618059a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:11.255946 31540 consensus_queue.cc:260] T 6a5f99276dad456ea80e3c9f9618059a P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:11.256438 31540 ts_tablet_manager.cc:1398] T 6a5f99276dad456ea80e3c9f9618059a P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:11.257475 31544 tablet_bootstrap.cc:492] T c4851258508d46c88493503a6257f365 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:11.259796 31544 tablet_bootstrap.cc:654] T c4851258508d46c88493503a6257f365 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:11.260785 27617 catalog_manager.cc:5484] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:11.268581 31476 raft_consensus.cc:482] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:11.268770 31476 raft_consensus.cc:504] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:11.269403 31476 leader_election.cc:288] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:11.269843 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89badd2c37f4f6e888b86b0e4226bd8" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:11.269866 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89badd2c37f4f6e888b86b0e4226bd8" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:12:11.270179 27759 raft_consensus.cc:2454] T c89badd2c37f4f6e888b86b0e4226bd8 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
W20240501 08:12:11.270424 30739 leader_election.cc:341] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:11.270677 30739 leader_election.cc:302] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 64998323a1f4482d8704bf20594d13a5; no voters: a29cd590999b4f32849ff5689cd53c11
I20240501 08:12:11.270936 31476 raft_consensus.cc:2781] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:11.271065 31476 raft_consensus.cc:482] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:11.271131 31476 raft_consensus.cc:3037] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:11.273202 31476 raft_consensus.cc:504] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:11.273974 31476 leader_election.cc:288] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Requested vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:11.274418 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89badd2c37f4f6e888b86b0e4226bd8" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
W20240501 08:12:11.274922 30739 leader_election.cc:341] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:11.274924 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89badd2c37f4f6e888b86b0e4226bd8" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:12:11.275143 27759 raft_consensus.cc:3037] T c89badd2c37f4f6e888b86b0e4226bd8 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:11.277141 27759 raft_consensus.cc:2454] T c89badd2c37f4f6e888b86b0e4226bd8 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:11.277617 30739 leader_election.cc:302] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 64998323a1f4482d8704bf20594d13a5; no voters: a29cd590999b4f32849ff5689cd53c11
I20240501 08:12:11.278025 31476 raft_consensus.cc:2781] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:11.278216 31476 raft_consensus.cc:686] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Becoming Leader. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Running, Role: LEADER
I20240501 08:12:11.278685 31544 tablet_bootstrap.cc:492] T c4851258508d46c88493503a6257f365 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:11.278640 31476 consensus_queue.cc:237] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:11.278865 31544 ts_tablet_manager.cc:1367] T c4851258508d46c88493503a6257f365 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.002s
I20240501 08:12:11.279685 31544 raft_consensus.cc:348] T c4851258508d46c88493503a6257f365 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:11.279919 31544 raft_consensus.cc:374] T c4851258508d46c88493503a6257f365 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:11.280017 31544 raft_consensus.cc:729] T c4851258508d46c88493503a6257f365 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:11.280269 31544 consensus_queue.cc:260] T c4851258508d46c88493503a6257f365 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:11.280773 31544 ts_tablet_manager.cc:1398] T c4851258508d46c88493503a6257f365 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:11.281222 31544 tablet_bootstrap.cc:492] T c89badd2c37f4f6e888b86b0e4226bd8 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:11.282444 27613 catalog_manager.cc:5484] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64998323a1f4482d8704bf20594d13a5 (127.21.84.135). New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:11.283789 31544 tablet_bootstrap.cc:654] T c89badd2c37f4f6e888b86b0e4226bd8 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:11.286232 31544 tablet_bootstrap.cc:492] T c89badd2c37f4f6e888b86b0e4226bd8 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:11.286392 31544 ts_tablet_manager.cc:1367] T c89badd2c37f4f6e888b86b0e4226bd8 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 08:12:11.287151 31544 raft_consensus.cc:348] T c89badd2c37f4f6e888b86b0e4226bd8 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:11.287356 31544 raft_consensus.cc:374] T c89badd2c37f4f6e888b86b0e4226bd8 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:11.287443 31544 raft_consensus.cc:729] T c89badd2c37f4f6e888b86b0e4226bd8 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:11.287662 31544 consensus_queue.cc:260] T c89badd2c37f4f6e888b86b0e4226bd8 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:11.288189 31544 ts_tablet_manager.cc:1398] T c89badd2c37f4f6e888b86b0e4226bd8 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:11.288954 31523 raft_consensus.cc:482] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:11.289103 31523 raft_consensus.cc:504] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:11.289973 31523 leader_election.cc:288] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:11.290417 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8277d70d49f54197af60e0be86d34a9b" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:11.290511 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8277d70d49f54197af60e0be86d34a9b" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:11.290691 30804 raft_consensus.cc:2454] T 8277d70d49f54197af60e0be86d34a9b P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:11.290760 30536 raft_consensus.cc:2454] T 8277d70d49f54197af60e0be86d34a9b P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:11.291061 30871 leader_election.cc:302] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:11.291322 31523 raft_consensus.cc:2781] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:11.291404 31523 raft_consensus.cc:482] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:11.291517 31523 raft_consensus.cc:3037] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:11.293329 31523 raft_consensus.cc:504] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:11.293920 31523 leader_election.cc:288] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Requested vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:11.294282 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8277d70d49f54197af60e0be86d34a9b" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:11.294476 30804 raft_consensus.cc:3037] T 8277d70d49f54197af60e0be86d34a9b P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:11.294416 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8277d70d49f54197af60e0be86d34a9b" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:11.294605 30536 raft_consensus.cc:3037] T 8277d70d49f54197af60e0be86d34a9b P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:11.297658 30536 raft_consensus.cc:2454] T 8277d70d49f54197af60e0be86d34a9b P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:11.298384 30804 raft_consensus.cc:2454] T 8277d70d49f54197af60e0be86d34a9b P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:11.298861 30873 leader_election.cc:302] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:11.299544 31523 raft_consensus.cc:2781] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:11.299932 31523 raft_consensus.cc:686] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Becoming Leader. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:12:11.300542 31523 consensus_queue.cc:237] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:11.303737 27620 catalog_manager.cc:5484] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 reported cstate change: term changed from 0 to 1, leader changed from <none> to a29cd590999b4f32849ff5689cd53c11 (127.21.84.136). New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:11.390677 27617 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:11.398381 27759 raft_consensus.cc:1261] T c89badd2c37f4f6e888b86b0e4226bd8 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:11.399104 30936 raft_consensus.cc:3037] T c89badd2c37f4f6e888b86b0e4226bd8 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:11.402141 30936 raft_consensus.cc:1261] T c89badd2c37f4f6e888b86b0e4226bd8 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:11.405723 31458 consensus_queue.cc:1035] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:11.406687 31562 consensus_queue.cc:1035] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:11.410861 27620 catalog_manager.cc:5576] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845) does not have the latest schema for tablet c89badd2c37f4f6e888b86b0e4226bd8 (table rebalancer_dynamic_table [id=315b3679582b4b608aff3d4852535612]). Expected version 1 got 0
I20240501 08:12:11.422313 31566 tablet.cc:1722] T c89badd2c37f4f6e888b86b0e4226bd8 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:11.423236 31565 tablet.cc:1722] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:11.424007 31566 tablet.cc:1620] T c89badd2c37f4f6e888b86b0e4226bd8 P a29cd590999b4f32849ff5689cd53c11: MemRowSet was empty: no flush needed.
I20240501 08:12:11.424007 31565 tablet.cc:1620] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5: MemRowSet was empty: no flush needed.
I20240501 08:12:11.425448 27617 catalog_manager.cc:5576] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503) does not have the latest schema for tablet c89badd2c37f4f6e888b86b0e4226bd8 (table rebalancer_dynamic_table [id=315b3679582b4b608aff3d4852535612]). Expected version 1 got 0
I20240501 08:12:11.426738 27620 catalog_manager.cc:5576] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845) does not have the latest schema for tablet c89badd2c37f4f6e888b86b0e4226bd8 (table rebalancer_dynamic_table [id=315b3679582b4b608aff3d4852535612]). Expected version 1 got 0
I20240501 08:12:11.433228 27617 catalog_manager.cc:5576] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471) does not have the latest schema for tablet c89badd2c37f4f6e888b86b0e4226bd8 (table rebalancer_dynamic_table [id=315b3679582b4b608aff3d4852535612]). Expected version 1 got 0
I20240501 08:12:11.435451 27620 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=315b3679582b4b608aff3d4852535612] alter complete (version 1)
I20240501 08:12:11.437886 31568 tablet.cc:1722] T c89badd2c37f4f6e888b86b0e4226bd8 P 6331f77dae9d452184e8cd0cbf5f1b43: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:11.438520 31568 tablet.cc:1620] T c89badd2c37f4f6e888b86b0e4226bd8 P 6331f77dae9d452184e8cd0cbf5f1b43: MemRowSet was empty: no flush needed.
I20240501 08:12:11.439589 27617 catalog_manager.cc:5576] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471) does not have the latest schema for tablet c89badd2c37f4f6e888b86b0e4226bd8 (table rebalancer_dynamic_table [id=315b3679582b4b608aff3d4852535612]). Expected version 1 got 0
I20240501 08:12:11.538175 27617 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:11.545264 31565 tablet.cc:1722] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:11.545775 31565 tablet.cc:1620] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5: MemRowSet was empty: no flush needed.
I20240501 08:12:11.546949 31568 tablet.cc:1722] T c89badd2c37f4f6e888b86b0e4226bd8 P 6331f77dae9d452184e8cd0cbf5f1b43: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:11.547914 31568 tablet.cc:1620] T c89badd2c37f4f6e888b86b0e4226bd8 P 6331f77dae9d452184e8cd0cbf5f1b43: MemRowSet was empty: no flush needed.
I20240501 08:12:11.552927 31566 tablet.cc:1722] T c89badd2c37f4f6e888b86b0e4226bd8 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:11.553467 31566 tablet.cc:1620] T c89badd2c37f4f6e888b86b0e4226bd8 P a29cd590999b4f32849ff5689cd53c11: MemRowSet was empty: no flush needed.
I20240501 08:12:11.556294 27617 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=315b3679582b4b608aff3d4852535612] alter complete (version 2)
I20240501 08:12:11.570369 31562 raft_consensus.cc:482] T 6a5f99276dad456ea80e3c9f9618059a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:11.570581 31562 raft_consensus.cc:504] T 6a5f99276dad456ea80e3c9f9618059a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:11.571326 31562 leader_election.cc:288] T 6a5f99276dad456ea80e3c9f9618059a P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:11.571751 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5f99276dad456ea80e3c9f9618059a" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:12:11.572038 27890 raft_consensus.cc:2379] T 6a5f99276dad456ea80e3c9f9618059a P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 64998323a1f4482d8704bf20594d13a5 in current term 1: Already voted for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in this term.
I20240501 08:12:11.572149 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5f99276dad456ea80e3c9f9618059a" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:12:11.572628 30739 leader_election.cc:302] T 6a5f99276dad456ea80e3c9f9618059a P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5; no voters: 6331f77dae9d452184e8cd0cbf5f1b43, f1525cbc82a24d13b65f136695c903af
I20240501 08:12:11.572890 31562 raft_consensus.cc:3037] T 6a5f99276dad456ea80e3c9f9618059a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:11.574621 31497 raft_consensus.cc:482] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:11.574807 31562 raft_consensus.cc:2726] T 6a5f99276dad456ea80e3c9f9618059a P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:11.574807 31497 raft_consensus.cc:504] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:11.575426 31497 leader_election.cc:288] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:11.575855 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4851258508d46c88493503a6257f365" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:11.575917 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4851258508d46c88493503a6257f365" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:11.576061 30804 raft_consensus.cc:2454] T c4851258508d46c88493503a6257f365 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:12:11.576151 30936 raft_consensus.cc:2454] T c4851258508d46c88493503a6257f365 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:12:11.576894 27826 leader_election.cc:302] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:11.577195 31497 raft_consensus.cc:2781] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:11.577289 31497 raft_consensus.cc:482] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:11.577363 31497 raft_consensus.cc:3037] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:11.579330 31497 raft_consensus.cc:504] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:11.579954 31497 leader_election.cc:288] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Requested vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:11.580332 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4851258508d46c88493503a6257f365" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:11.580341 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4851258508d46c88493503a6257f365" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:11.580497 30804 raft_consensus.cc:3037] T c4851258508d46c88493503a6257f365 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:11.580529 30936 raft_consensus.cc:3037] T c4851258508d46c88493503a6257f365 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:11.582414 30804 raft_consensus.cc:2454] T c4851258508d46c88493503a6257f365 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:12:11.582490 30936 raft_consensus.cc:2454] T c4851258508d46c88493503a6257f365 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:12:11.582808 27826 leader_election.cc:302] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:11.583101 31497 raft_consensus.cc:2781] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:11.583258 31497 raft_consensus.cc:686] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Becoming Leader. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Running, Role: LEADER
I20240501 08:12:11.583515 31497 consensus_queue.cc:237] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:11.586701 27617 catalog_manager.cc:5484] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af reported cstate change: term changed from 0 to 1, leader changed from <none> to f1525cbc82a24d13b65f136695c903af (127.21.84.130). New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:11.661046 27617 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:11.668787 31565 tablet.cc:1722] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:11.669588 31566 tablet.cc:1722] T c89badd2c37f4f6e888b86b0e4226bd8 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:11.671002 31568 tablet.cc:1722] T c89badd2c37f4f6e888b86b0e4226bd8 P 6331f77dae9d452184e8cd0cbf5f1b43: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:11.678062 27617 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table_new_name [id=315b3679582b4b608aff3d4852535612] alter complete (version 3)
I20240501 08:12:11.772164 31551 consensus_queue.cc:1035] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:11.772775 31457 consensus_queue.cc:1035] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:11.779105 31176 consensus_queue.cc:1035] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:11.781077 31523 consensus_queue.cc:1035] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:11.784811 27611 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:11.785154 27611 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:11.788462 27611 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet c89badd2c37f4f6e888b86b0e4226bd8
I20240501 08:12:11.789786 30917 tablet_service.cc:1514] Processing DeleteTablet for tablet c89badd2c37f4f6e888b86b0e4226bd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:11 UTC) from {username='slave'} at 127.0.0.1:54036
I20240501 08:12:11.789822 27741 tablet_service.cc:1514] Processing DeleteTablet for tablet c89badd2c37f4f6e888b86b0e4226bd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:11 UTC) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:11.790026 30784 tablet_service.cc:1514] Processing DeleteTablet for tablet c89badd2c37f4f6e888b86b0e4226bd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:11 UTC) from {username='slave'} at 127.0.0.1:38952
I20240501 08:12:11.793615 31575 tablet_replica.cc:330] T c89badd2c37f4f6e888b86b0e4226bd8 P a29cd590999b4f32849ff5689cd53c11: stopping tablet replica
I20240501 08:12:11.793910 31575 raft_consensus.cc:2229] T c89badd2c37f4f6e888b86b0e4226bd8 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:11.794271 31575 raft_consensus.cc:2258] T c89badd2c37f4f6e888b86b0e4226bd8 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:11.795291 31577 tablet_replica.cc:330] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5: stopping tablet replica
I20240501 08:12:11.795387 31575 ts_tablet_manager.cc:1875] T c89badd2c37f4f6e888b86b0e4226bd8 P a29cd590999b4f32849ff5689cd53c11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:11.795584 31577 raft_consensus.cc:2229] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:11.796013 31577 raft_consensus.cc:2258] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:11.796516 31576 tablet_replica.cc:330] T c89badd2c37f4f6e888b86b0e4226bd8 P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:11.796772 31576 raft_consensus.cc:2229] T c89badd2c37f4f6e888b86b0e4226bd8 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:11.796821 31577 ts_tablet_manager.cc:1875] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:11.797058 31576 raft_consensus.cc:2258] T c89badd2c37f4f6e888b86b0e4226bd8 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:11.797989 31576 ts_tablet_manager.cc:1875] T c89badd2c37f4f6e888b86b0e4226bd8 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:11.803088 31577 ts_tablet_manager.cc:1888] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:11.803233 31577 log.cc:1196] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal/wals/c89badd2c37f4f6e888b86b0e4226bd8
I20240501 08:12:11.804075 31577 ts_tablet_manager.cc:1909] T c89badd2c37f4f6e888b86b0e4226bd8 P 64998323a1f4482d8704bf20594d13a5: Deleting consensus metadata
I20240501 08:12:11.805250 27605 catalog_manager.cc:4830] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): tablet c89badd2c37f4f6e888b86b0e4226bd8 (table rebalancer_dynamic_table_new_name [id=315b3679582b4b608aff3d4852535612]) successfully deleted
I20240501 08:12:11.806488 31576 ts_tablet_manager.cc:1888] T c89badd2c37f4f6e888b86b0e4226bd8 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:11.806625 31576 log.cc:1196] T c89badd2c37f4f6e888b86b0e4226bd8 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/c89badd2c37f4f6e888b86b0e4226bd8
I20240501 08:12:11.807116 31576 ts_tablet_manager.cc:1909] T c89badd2c37f4f6e888b86b0e4226bd8 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting consensus metadata
I20240501 08:12:11.807803 31575 ts_tablet_manager.cc:1888] T c89badd2c37f4f6e888b86b0e4226bd8 P a29cd590999b4f32849ff5689cd53c11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:11.807930 31575 log.cc:1196] T c89badd2c37f4f6e888b86b0e4226bd8 P a29cd590999b4f32849ff5689cd53c11: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/wals/c89badd2c37f4f6e888b86b0e4226bd8
I20240501 08:12:11.808158 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet c89badd2c37f4f6e888b86b0e4226bd8 (table rebalancer_dynamic_table_new_name [id=315b3679582b4b608aff3d4852535612]) successfully deleted
I20240501 08:12:11.808568 31575 ts_tablet_manager.cc:1909] T c89badd2c37f4f6e888b86b0e4226bd8 P a29cd590999b4f32849ff5689cd53c11: Deleting consensus metadata
I20240501 08:12:11.809403 27606 catalog_manager.cc:4830] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): tablet c89badd2c37f4f6e888b86b0e4226bd8 (table rebalancer_dynamic_table_new_name [id=315b3679582b4b608aff3d4852535612]) successfully deleted
I20240501 08:12:11.892958 27613 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:11.900272 30652 tablet_service.cc:1467] Processing CreateTablet for tablet d4244c394dd44477a1bcf5d991cb3c52 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=905b4af72cfb4f96b1f4571474f9759c]), partition=
I20240501 08:12:11.900511 30784 tablet_service.cc:1467] Processing CreateTablet for tablet d4244c394dd44477a1bcf5d991cb3c52 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=905b4af72cfb4f96b1f4571474f9759c]), partition=
I20240501 08:12:11.900738 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4244c394dd44477a1bcf5d991cb3c52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.900849 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4244c394dd44477a1bcf5d991cb3c52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.901072 28132 tablet_service.cc:1467] Processing CreateTablet for tablet d4244c394dd44477a1bcf5d991cb3c52 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=905b4af72cfb4f96b1f4571474f9759c]), partition=
I20240501 08:12:11.901336 28132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4244c394dd44477a1bcf5d991cb3c52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:11.909422 31580 tablet_bootstrap.cc:492] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:11.909893 31579 tablet_bootstrap.cc:492] T d4244c394dd44477a1bcf5d991cb3c52 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:11.910504 31581 tablet_bootstrap.cc:492] T d4244c394dd44477a1bcf5d991cb3c52 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:11.911550 31579 tablet_bootstrap.cc:654] T d4244c394dd44477a1bcf5d991cb3c52 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:11.911926 31580 tablet_bootstrap.cc:654] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:11.912132 31581 tablet_bootstrap.cc:654] T d4244c394dd44477a1bcf5d991cb3c52 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:11.914182 31579 tablet_bootstrap.cc:492] T d4244c394dd44477a1bcf5d991cb3c52 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:12:11.914326 31580 tablet_bootstrap.cc:492] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:11.914351 31579 ts_tablet_manager.cc:1367] T d4244c394dd44477a1bcf5d991cb3c52 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240501 08:12:11.914454 31580 ts_tablet_manager.cc:1367] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 08:12:11.915125 31580 raft_consensus.cc:348] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:11.915171 31579 raft_consensus.cc:348] T d4244c394dd44477a1bcf5d991cb3c52 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:11.915341 31580 raft_consensus.cc:374] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:11.915383 31579 raft_consensus.cc:374] T d4244c394dd44477a1bcf5d991cb3c52 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:11.915417 31580 raft_consensus.cc:729] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:11.915474 31579 raft_consensus.cc:729] T d4244c394dd44477a1bcf5d991cb3c52 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:12:11.915633 31580 consensus_queue.cc:260] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:11.915701 31579 consensus_queue.cc:260] T d4244c394dd44477a1bcf5d991cb3c52 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:11.916262 31579 ts_tablet_manager.cc:1398] T d4244c394dd44477a1bcf5d991cb3c52 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:11.917357 31580 ts_tablet_manager.cc:1398] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:11.922302 31581 tablet_bootstrap.cc:492] T d4244c394dd44477a1bcf5d991cb3c52 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:11.922575 31581 ts_tablet_manager.cc:1367] T d4244c394dd44477a1bcf5d991cb3c52 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240501 08:12:11.923274 31581 raft_consensus.cc:348] T d4244c394dd44477a1bcf5d991cb3c52 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:11.923456 31581 raft_consensus.cc:374] T d4244c394dd44477a1bcf5d991cb3c52 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:11.923559 31581 raft_consensus.cc:729] T d4244c394dd44477a1bcf5d991cb3c52 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:11.923774 31581 consensus_queue.cc:260] T d4244c394dd44477a1bcf5d991cb3c52 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:11.924382 31581 ts_tablet_manager.cc:1398] T d4244c394dd44477a1bcf5d991cb3c52 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:11.990796 31538 raft_consensus.cc:482] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:11.991026 31538 raft_consensus.cc:504] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:11.991911 31538 leader_election.cc:288] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:11.992328 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4244c394dd44477a1bcf5d991cb3c52" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:11.992597 30804 raft_consensus.cc:2454] T d4244c394dd44477a1bcf5d991cb3c52 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 0.
I20240501 08:12:11.993132 30606 leader_election.cc:302] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:11.993432 31538 raft_consensus.cc:2781] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:11.993559 31538 raft_consensus.cc:482] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:11.993674 31538 raft_consensus.cc:3037] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:11.995955 31538 raft_consensus.cc:504] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:11.996639 31538 leader_election.cc:288] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 election: Requested vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:11.997053 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4244c394dd44477a1bcf5d991cb3c52" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:11.997249 30804 raft_consensus.cc:3037] T d4244c394dd44477a1bcf5d991cb3c52 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:11.999590 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4244c394dd44477a1bcf5d991cb3c52" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:12:11.999665 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4244c394dd44477a1bcf5d991cb3c52" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:12:11.999832 28151 raft_consensus.cc:2454] T d4244c394dd44477a1bcf5d991cb3c52 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 0.
I20240501 08:12:11.999918 30804 raft_consensus.cc:2454] T d4244c394dd44477a1bcf5d991cb3c52 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 1.
I20240501 08:12:12.000342 30606 leader_election.cc:302] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:12.000723 31538 raft_consensus.cc:2781] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:12.001231 31538 raft_consensus.cc:686] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Becoming Leader. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Running, Role: LEADER
I20240501 08:12:12.001587 31538 consensus_queue.cc:237] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:12.004802 27617 catalog_manager.cc:5484] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134). New cstate: current_term: 1 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:12.113713 27617 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:12.120038 28151 raft_consensus.cc:3037] T d4244c394dd44477a1bcf5d991cb3c52 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.120093 30804 raft_consensus.cc:1261] T d4244c394dd44477a1bcf5d991cb3c52 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:12.120666 31587 consensus_queue.cc:1035] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:12.123019 28151 raft_consensus.cc:1261] T d4244c394dd44477a1bcf5d991cb3c52 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:12.123579 31587 consensus_queue.cc:1035] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:12.124589 27617 catalog_manager.cc:5576] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503) does not have the latest schema for tablet d4244c394dd44477a1bcf5d991cb3c52 (table rebalancer_dynamic_table [id=905b4af72cfb4f96b1f4571474f9759c]). Expected version 1 got 0
I20240501 08:12:12.125761 31565 tablet.cc:1722] T d4244c394dd44477a1bcf5d991cb3c52 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:12.126282 31565 tablet.cc:1620] T d4244c394dd44477a1bcf5d991cb3c52 P 64998323a1f4482d8704bf20594d13a5: MemRowSet was empty: no flush needed.
I20240501 08:12:12.126330 31593 tablet.cc:1722] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:12.126739 27617 catalog_manager.cc:5576] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933) does not have the latest schema for tablet d4244c394dd44477a1bcf5d991cb3c52 (table rebalancer_dynamic_table [id=905b4af72cfb4f96b1f4571474f9759c]). Expected version 1 got 0
I20240501 08:12:12.128584 31593 tablet.cc:1620] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:12.131135 27617 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=905b4af72cfb4f96b1f4571474f9759c] alter complete (version 1)
I20240501 08:12:12.140542 31596 tablet.cc:1722] T d4244c394dd44477a1bcf5d991cb3c52 P 91d11eae5ef6432ca86eb8ed446b6ea6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:12.141312 31596 tablet.cc:1620] T d4244c394dd44477a1bcf5d991cb3c52 P 91d11eae5ef6432ca86eb8ed446b6ea6: MemRowSet was empty: no flush needed.
I20240501 08:12:12.141590 27613 catalog_manager.cc:5576] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323) does not have the latest schema for tablet d4244c394dd44477a1bcf5d991cb3c52 (table rebalancer_dynamic_table [id=905b4af72cfb4f96b1f4571474f9759c]). Expected version 1 got 0
I20240501 08:12:12.173336 31497 consensus_queue.cc:1035] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:12.175338 31497 consensus_queue.cc:1035] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:12.196175 27613 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_extra_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:12.206365 27872 tablet_service.cc:1467] Processing CreateTablet for tablet d6bb70263134414b9945a87087b55a2c (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=8b4eeb92acc446a29303f629cf185156]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:12.206382 30517 tablet_service.cc:1467] Processing CreateTablet for tablet d6bb70263134414b9945a87087b55a2c (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=8b4eeb92acc446a29303f629cf185156]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:12.206411 30784 tablet_service.cc:1467] Processing CreateTablet for tablet 98525304f2334bf0a3f52c06df58b3ba (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=8b4eeb92acc446a29303f629cf185156]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:12.206851 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6bb70263134414b9945a87087b55a2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:12.206857 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98525304f2334bf0a3f52c06df58b3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:12.206864 27872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6bb70263134414b9945a87087b55a2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:12.211517 31581 tablet_bootstrap.cc:492] T 98525304f2334bf0a3f52c06df58b3ba P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:12.213192 31581 tablet_bootstrap.cc:654] T 98525304f2334bf0a3f52c06df58b3ba P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:12.213634 30917 tablet_service.cc:1467] Processing CreateTablet for tablet 98525304f2334bf0a3f52c06df58b3ba (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=8b4eeb92acc446a29303f629cf185156]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:12.214128 30917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98525304f2334bf0a3f52c06df58b3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:12.205890 30652 tablet_service.cc:1467] Processing CreateTablet for tablet d6bb70263134414b9945a87087b55a2c (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=8b4eeb92acc446a29303f629cf185156]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:12.214592 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6bb70263134414b9945a87087b55a2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:12.219715 28132 tablet_service.cc:1467] Processing CreateTablet for tablet 98525304f2334bf0a3f52c06df58b3ba (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=8b4eeb92acc446a29303f629cf185156]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:12.220384 28132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98525304f2334bf0a3f52c06df58b3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:12.220528 30651 tablet_service.cc:1467] Processing CreateTablet for tablet 66b8b9c025b941a1a1b6e681fa0bbe90 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=8b4eeb92acc446a29303f629cf185156]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:12.220672 30782 tablet_service.cc:1467] Processing CreateTablet for tablet 66b8b9c025b941a1a1b6e681fa0bbe90 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=8b4eeb92acc446a29303f629cf185156]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:12.220890 30651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66b8b9c025b941a1a1b6e681fa0bbe90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:12.221019 30782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66b8b9c025b941a1a1b6e681fa0bbe90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:12.224197 27741 tablet_service.cc:1467] Processing CreateTablet for tablet 66b8b9c025b941a1a1b6e681fa0bbe90 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=8b4eeb92acc446a29303f629cf185156]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:12.224764 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66b8b9c025b941a1a1b6e681fa0bbe90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:12.243796 27611 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:12.246464 31580 tablet_bootstrap.cc:492] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:12.248791 31580 tablet_bootstrap.cc:654] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:12.252746 31601 tablet_bootstrap.cc:492] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:12.255497 31601 tablet_bootstrap.cc:654] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:12.260987 31608 tablet_bootstrap.cc:492] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:12.261652 31593 tablet.cc:1722] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:12.262476 31593 tablet.cc:1620] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:12.263446 31580 tablet_bootstrap.cc:492] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:12.263648 31580 ts_tablet_manager.cc:1367] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.005s
I20240501 08:12:12.264451 31602 tablet_bootstrap.cc:492] T d6bb70263134414b9945a87087b55a2c P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:12.264525 31608 tablet_bootstrap.cc:654] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:12.265296 31580 raft_consensus.cc:348] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:12.265547 31580 raft_consensus.cc:374] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:12.265632 31580 raft_consensus.cc:729] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:12.266067 31580 consensus_queue.cc:260] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:12.266758 31580 ts_tablet_manager.cc:1398] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240501 08:12:12.267364 31580 tablet_bootstrap.cc:492] T d6bb70263134414b9945a87087b55a2c P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:12.268477 31602 tablet_bootstrap.cc:654] T d6bb70263134414b9945a87087b55a2c P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:12.271149 31565 tablet.cc:1722] T d4244c394dd44477a1bcf5d991cb3c52 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:12.271791 31565 tablet.cc:1620] T d4244c394dd44477a1bcf5d991cb3c52 P 64998323a1f4482d8704bf20594d13a5: MemRowSet was empty: no flush needed.
I20240501 08:12:12.271910 31580 tablet_bootstrap.cc:654] T d6bb70263134414b9945a87087b55a2c P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:12.275202 31602 tablet_bootstrap.cc:492] T d6bb70263134414b9945a87087b55a2c P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:12.275408 31602 ts_tablet_manager.cc:1367] T d6bb70263134414b9945a87087b55a2c P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240501 08:12:12.276330 31602 raft_consensus.cc:348] T d6bb70263134414b9945a87087b55a2c P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:12.279994 31602 raft_consensus.cc:374] T d6bb70263134414b9945a87087b55a2c P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:12.280126 31602 raft_consensus.cc:729] T d6bb70263134414b9945a87087b55a2c P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:12.280730 31580 tablet_bootstrap.cc:492] T d6bb70263134414b9945a87087b55a2c P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:12.280928 31580 ts_tablet_manager.cc:1367] T d6bb70263134414b9945a87087b55a2c P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.009s
I20240501 08:12:12.281337 31602 consensus_queue.cc:260] T d6bb70263134414b9945a87087b55a2c P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:12.283879 31602 ts_tablet_manager.cc:1398] T d6bb70263134414b9945a87087b55a2c P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 08:12:12.285643 27611 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=905b4af72cfb4f96b1f4571474f9759c] alter complete (version 2)
I20240501 08:12:12.286617 31580 raft_consensus.cc:348] T d6bb70263134414b9945a87087b55a2c P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:12.286916 31580 raft_consensus.cc:374] T d6bb70263134414b9945a87087b55a2c P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:12.287089 31580 raft_consensus.cc:729] T d6bb70263134414b9945a87087b55a2c P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:12.288601 31580 consensus_queue.cc:260] T d6bb70263134414b9945a87087b55a2c P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:12.289402 31608 tablet_bootstrap.cc:492] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:12:12.289387 31580 ts_tablet_manager.cc:1398] T d6bb70263134414b9945a87087b55a2c P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20240501 08:12:12.289829 31608 ts_tablet_manager.cc:1367] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.009s
I20240501 08:12:12.290527 31608 raft_consensus.cc:348] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:12.290688 31608 raft_consensus.cc:374] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:12.290786 31608 raft_consensus.cc:729] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:12:12.291000 31608 consensus_queue.cc:260] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:12.291476 31608 ts_tablet_manager.cc:1398] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:12:12.302852 31579 tablet_bootstrap.cc:492] T 98525304f2334bf0a3f52c06df58b3ba P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:12.305052 31579 tablet_bootstrap.cc:654] T 98525304f2334bf0a3f52c06df58b3ba P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:12.306237 31596 tablet.cc:1722] T d4244c394dd44477a1bcf5d991cb3c52 P 91d11eae5ef6432ca86eb8ed446b6ea6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:12.306752 31596 tablet.cc:1620] T d4244c394dd44477a1bcf5d991cb3c52 P 91d11eae5ef6432ca86eb8ed446b6ea6: MemRowSet was empty: no flush needed.
I20240501 08:12:12.307077 31579 tablet_bootstrap.cc:492] T 98525304f2334bf0a3f52c06df58b3ba P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:12:12.307236 31579 ts_tablet_manager.cc:1367] T 98525304f2334bf0a3f52c06df58b3ba P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20240501 08:12:12.308962 31579 raft_consensus.cc:348] T 98525304f2334bf0a3f52c06df58b3ba P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:12.309301 31579 raft_consensus.cc:374] T 98525304f2334bf0a3f52c06df58b3ba P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:12.309389 31579 raft_consensus.cc:729] T 98525304f2334bf0a3f52c06df58b3ba P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:12:12.309764 31579 consensus_queue.cc:260] T 98525304f2334bf0a3f52c06df58b3ba P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:12.310417 31579 ts_tablet_manager.cc:1398] T 98525304f2334bf0a3f52c06df58b3ba P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 08:12:12.325582 31581 tablet_bootstrap.cc:492] T 98525304f2334bf0a3f52c06df58b3ba P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:12.325980 31581 ts_tablet_manager.cc:1367] T 98525304f2334bf0a3f52c06df58b3ba P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.114s user 0.008s sys 0.000s
I20240501 08:12:12.326896 31581 raft_consensus.cc:348] T 98525304f2334bf0a3f52c06df58b3ba P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:12.328382 31581 raft_consensus.cc:374] T 98525304f2334bf0a3f52c06df58b3ba P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:12.328575 31581 raft_consensus.cc:729] T 98525304f2334bf0a3f52c06df58b3ba P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:12.329030 31581 consensus_queue.cc:260] T 98525304f2334bf0a3f52c06df58b3ba P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:12.333456 31604 tablet_bootstrap.cc:492] T 98525304f2334bf0a3f52c06df58b3ba P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:12.334676 31581 ts_tablet_manager.cc:1398] T 98525304f2334bf0a3f52c06df58b3ba P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240501 08:12:12.335326 31601 tablet_bootstrap.cc:492] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:12:12.335494 31601 ts_tablet_manager.cc:1367] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.083s user 0.023s sys 0.012s
I20240501 08:12:12.335901 31604 tablet_bootstrap.cc:654] T 98525304f2334bf0a3f52c06df58b3ba P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:12.337747 31601 raft_consensus.cc:348] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:12.338027 31601 raft_consensus.cc:374] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:12.338073 31581 tablet_bootstrap.cc:492] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:12.338172 31601 raft_consensus.cc:729] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:12:12.338552 31601 consensus_queue.cc:260] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:12.340395 31581 tablet_bootstrap.cc:654] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:12.341526 31601 ts_tablet_manager.cc:1398] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:12:12.344918 31604 tablet_bootstrap.cc:492] T 98525304f2334bf0a3f52c06df58b3ba P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:12.345162 31604 ts_tablet_manager.cc:1367] T 98525304f2334bf0a3f52c06df58b3ba P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20240501 08:12:12.346087 31604 raft_consensus.cc:348] T 98525304f2334bf0a3f52c06df58b3ba P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:12.346398 31604 raft_consensus.cc:374] T 98525304f2334bf0a3f52c06df58b3ba P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:12.346555 31604 raft_consensus.cc:729] T 98525304f2334bf0a3f52c06df58b3ba P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:12.346838 31604 consensus_queue.cc:260] T 98525304f2334bf0a3f52c06df58b3ba P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:12.347438 31604 ts_tablet_manager.cc:1398] T 98525304f2334bf0a3f52c06df58b3ba P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:12.352172 31581 tablet_bootstrap.cc:492] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:12.352335 31581 ts_tablet_manager.cc:1367] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.006s
I20240501 08:12:12.353111 31581 raft_consensus.cc:348] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:12.353320 31581 raft_consensus.cc:374] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:12.353417 31581 raft_consensus.cc:729] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:12.353678 31581 consensus_queue.cc:260] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:12.354789 31581 ts_tablet_manager.cc:1398] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:12.397540 27613 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:12.405830 31593 tablet.cc:1722] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:12.406525 31596 tablet.cc:1722] T d4244c394dd44477a1bcf5d991cb3c52 P 91d11eae5ef6432ca86eb8ed446b6ea6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:12.414235 31565 tablet.cc:1722] T d4244c394dd44477a1bcf5d991cb3c52 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:12.418406 27613 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table_new_name [id=905b4af72cfb4f96b1f4571474f9759c] alter complete (version 3)
I20240501 08:12:12.440254 31616 raft_consensus.cc:482] T 98525304f2334bf0a3f52c06df58b3ba P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:12.440419 31616 raft_consensus.cc:504] T 98525304f2334bf0a3f52c06df58b3ba P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:12.441224 31616 leader_election.cc:288] T 98525304f2334bf0a3f52c06df58b3ba P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:12.441658 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98525304f2334bf0a3f52c06df58b3ba" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:12.441915 30803 raft_consensus.cc:2454] T 98525304f2334bf0a3f52c06df58b3ba P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:12.442390 30871 leader_election.cc:302] T 98525304f2334bf0a3f52c06df58b3ba P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:12.442708 31616 raft_consensus.cc:2781] T 98525304f2334bf0a3f52c06df58b3ba P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:12.442950 31616 raft_consensus.cc:482] T 98525304f2334bf0a3f52c06df58b3ba P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:12.443055 31616 raft_consensus.cc:3037] T 98525304f2334bf0a3f52c06df58b3ba P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.445890 31616 raft_consensus.cc:504] T 98525304f2334bf0a3f52c06df58b3ba P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:12.446511 31616 leader_election.cc:288] T 98525304f2334bf0a3f52c06df58b3ba P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Requested vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:12.447063 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98525304f2334bf0a3f52c06df58b3ba" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:12.447273 30803 raft_consensus.cc:3037] T 98525304f2334bf0a3f52c06df58b3ba P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.448896 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98525304f2334bf0a3f52c06df58b3ba" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:12:12.448908 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98525304f2334bf0a3f52c06df58b3ba" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:12:12.449177 28150 raft_consensus.cc:2454] T 98525304f2334bf0a3f52c06df58b3ba P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:12.450253 30803 raft_consensus.cc:2454] T 98525304f2334bf0a3f52c06df58b3ba P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:12.451009 30871 leader_election.cc:302] T 98525304f2334bf0a3f52c06df58b3ba P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, a29cd590999b4f32849ff5689cd53c11; no voters: 91d11eae5ef6432ca86eb8ed446b6ea6
I20240501 08:12:12.451310 31616 raft_consensus.cc:2781] T 98525304f2334bf0a3f52c06df58b3ba P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:12.451473 31616 raft_consensus.cc:686] T 98525304f2334bf0a3f52c06df58b3ba P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Becoming Leader. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:12:12.451818 31616 consensus_queue.cc:237] T 98525304f2334bf0a3f52c06df58b3ba P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:12.455273 27613 catalog_manager.cc:5484] T 98525304f2334bf0a3f52c06df58b3ba P a29cd590999b4f32849ff5689cd53c11 reported cstate change: term changed from 0 to 1, leader changed from <none> to a29cd590999b4f32849ff5689cd53c11 (127.21.84.136). New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:12.522312 27613 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:12.522573 27613 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:12.525561 27613 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet d4244c394dd44477a1bcf5d991cb3c52
I20240501 08:12:12.526463 30651 tablet_service.cc:1514] Processing DeleteTablet for tablet d4244c394dd44477a1bcf5d991cb3c52 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:12 UTC) from {username='slave'} at 127.0.0.1:57156
I20240501 08:12:12.526793 30784 tablet_service.cc:1514] Processing DeleteTablet for tablet d4244c394dd44477a1bcf5d991cb3c52 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:12 UTC) from {username='slave'} at 127.0.0.1:38952
I20240501 08:12:12.526896 28132 tablet_service.cc:1514] Processing DeleteTablet for tablet d4244c394dd44477a1bcf5d991cb3c52 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:12 UTC) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:12.527194 31619 tablet_replica.cc:330] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47: stopping tablet replica
I20240501 08:12:12.527420 31620 tablet_replica.cc:330] T d4244c394dd44477a1bcf5d991cb3c52 P 64998323a1f4482d8704bf20594d13a5: stopping tablet replica
I20240501 08:12:12.527441 31619 raft_consensus.cc:2229] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:12.527566 31621 tablet_replica.cc:330] T d4244c394dd44477a1bcf5d991cb3c52 P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:12.527623 31620 raft_consensus.cc:2229] T d4244c394dd44477a1bcf5d991cb3c52 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:12.527769 31621 raft_consensus.cc:2229] T d4244c394dd44477a1bcf5d991cb3c52 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:12.527828 31620 raft_consensus.cc:2258] T d4244c394dd44477a1bcf5d991cb3c52 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:12.527948 31619 raft_consensus.cc:2258] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:12.527989 31621 raft_consensus.cc:2258] T d4244c394dd44477a1bcf5d991cb3c52 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:12.528723 31620 ts_tablet_manager.cc:1875] T d4244c394dd44477a1bcf5d991cb3c52 P 64998323a1f4482d8704bf20594d13a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:12.528755 31621 ts_tablet_manager.cc:1875] T d4244c394dd44477a1bcf5d991cb3c52 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:12.528776 31619 ts_tablet_manager.cc:1875] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:12.534929 31621 ts_tablet_manager.cc:1888] T d4244c394dd44477a1bcf5d991cb3c52 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:12.535128 31621 log.cc:1196] T d4244c394dd44477a1bcf5d991cb3c52 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/d4244c394dd44477a1bcf5d991cb3c52
I20240501 08:12:12.535274 31619 ts_tablet_manager.cc:1888] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:12.535400 31619 log.cc:1196] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/wals/d4244c394dd44477a1bcf5d991cb3c52
I20240501 08:12:12.535555 31621 ts_tablet_manager.cc:1909] T d4244c394dd44477a1bcf5d991cb3c52 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting consensus metadata
I20240501 08:12:12.535895 31619 ts_tablet_manager.cc:1909] T d4244c394dd44477a1bcf5d991cb3c52 P 9a022e1a501f4d44a0468054d6c3de47: Deleting consensus metadata
I20240501 08:12:12.536101 31620 ts_tablet_manager.cc:1888] T d4244c394dd44477a1bcf5d991cb3c52 P 64998323a1f4482d8704bf20594d13a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:12.536258 31620 log.cc:1196] T d4244c394dd44477a1bcf5d991cb3c52 P 64998323a1f4482d8704bf20594d13a5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal/wals/d4244c394dd44477a1bcf5d991cb3c52
I20240501 08:12:12.536594 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet d4244c394dd44477a1bcf5d991cb3c52 (table rebalancer_dynamic_table_new_name [id=905b4af72cfb4f96b1f4571474f9759c]) successfully deleted
I20240501 08:12:12.536681 31620 ts_tablet_manager.cc:1909] T d4244c394dd44477a1bcf5d991cb3c52 P 64998323a1f4482d8704bf20594d13a5: Deleting consensus metadata
I20240501 08:12:12.537010 27604 catalog_manager.cc:4830] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): tablet d4244c394dd44477a1bcf5d991cb3c52 (table rebalancer_dynamic_table_new_name [id=905b4af72cfb4f96b1f4571474f9759c]) successfully deleted
I20240501 08:12:12.537536 27605 catalog_manager.cc:4830] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): tablet d4244c394dd44477a1bcf5d991cb3c52 (table rebalancer_dynamic_table_new_name [id=905b4af72cfb4f96b1f4571474f9759c]) successfully deleted
I20240501 08:12:12.596117 31599 raft_consensus.cc:482] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:12.596320 31599 raft_consensus.cc:504] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:12.597353 31599 leader_election.cc:288] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:12.597697 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6bb70263134414b9945a87087b55a2c" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:12.597959 30536 raft_consensus.cc:2454] T d6bb70263134414b9945a87087b55a2c P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:12:12.598524 27828 leader_election.cc:302] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:12.599216 31599 raft_consensus.cc:2781] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:12.599363 31599 raft_consensus.cc:482] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:12.599474 31599 raft_consensus.cc:3037] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.602324 31599 raft_consensus.cc:504] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:12.602931 31599 leader_election.cc:288] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Requested vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:12.603350 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6bb70263134414b9945a87087b55a2c" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:12.603534 30536 raft_consensus.cc:3037] T d6bb70263134414b9945a87087b55a2c P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.605429 30536 raft_consensus.cc:2454] T d6bb70263134414b9945a87087b55a2c P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:12:12.605885 27828 leader_election.cc:302] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:12.605926 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6bb70263134414b9945a87087b55a2c" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:12.606048 30671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6bb70263134414b9945a87087b55a2c" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:12.606220 31599 raft_consensus.cc:2781] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:12.606216 30672 raft_consensus.cc:2454] T d6bb70263134414b9945a87087b55a2c P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:12:12.606333 31599 raft_consensus.cc:686] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Becoming Leader. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Running, Role: LEADER
I20240501 08:12:12.606833 31599 consensus_queue.cc:237] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:12.610832 27613 catalog_manager.cc:5484] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af reported cstate change: term changed from 0 to 1, leader changed from <none> to f1525cbc82a24d13b65f136695c903af (127.21.84.130). New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:12.628109 27613 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:12.635022 30917 tablet_service.cc:1467] Processing CreateTablet for tablet a6ce2b2f52dc4bd3b04784e00721dbed (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cbddc790ca4147c6939b2b1169e03118]), partition=
I20240501 08:12:12.635214 27741 tablet_service.cc:1467] Processing CreateTablet for tablet a6ce2b2f52dc4bd3b04784e00721dbed (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cbddc790ca4147c6939b2b1169e03118]), partition=
I20240501 08:12:12.635298 30651 tablet_service.cc:1467] Processing CreateTablet for tablet a6ce2b2f52dc4bd3b04784e00721dbed (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cbddc790ca4147c6939b2b1169e03118]), partition=
I20240501 08:12:12.635490 30917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6ce2b2f52dc4bd3b04784e00721dbed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:12.635555 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6ce2b2f52dc4bd3b04784e00721dbed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:12.635638 30651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6ce2b2f52dc4bd3b04784e00721dbed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:12.640142 31608 tablet_bootstrap.cc:492] T a6ce2b2f52dc4bd3b04784e00721dbed P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:12.641686 31608 tablet_bootstrap.cc:654] T a6ce2b2f52dc4bd3b04784e00721dbed P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:12.641916 31604 tablet_bootstrap.cc:492] T a6ce2b2f52dc4bd3b04784e00721dbed P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:12.643213 31580 tablet_bootstrap.cc:492] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:12.643234 31604 tablet_bootstrap.cc:654] T a6ce2b2f52dc4bd3b04784e00721dbed P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:12.644480 31580 tablet_bootstrap.cc:654] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:12.645007 31604 tablet_bootstrap.cc:492] T a6ce2b2f52dc4bd3b04784e00721dbed P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:12.645164 31604 ts_tablet_manager.cc:1367] T a6ce2b2f52dc4bd3b04784e00721dbed P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:12.645799 31604 raft_consensus.cc:348] T a6ce2b2f52dc4bd3b04784e00721dbed P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:12.646065 31604 raft_consensus.cc:374] T a6ce2b2f52dc4bd3b04784e00721dbed P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:12.646150 31604 raft_consensus.cc:729] T a6ce2b2f52dc4bd3b04784e00721dbed P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:12.646468 31604 consensus_queue.cc:260] T a6ce2b2f52dc4bd3b04784e00721dbed P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:12.646806 31608 tablet_bootstrap.cc:492] T a6ce2b2f52dc4bd3b04784e00721dbed P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:12:12.646956 31608 ts_tablet_manager.cc:1367] T a6ce2b2f52dc4bd3b04784e00721dbed P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.002s
I20240501 08:12:12.647010 31604 ts_tablet_manager.cc:1398] T a6ce2b2f52dc4bd3b04784e00721dbed P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:12.647632 31608 raft_consensus.cc:348] T a6ce2b2f52dc4bd3b04784e00721dbed P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:12.647858 31608 raft_consensus.cc:374] T a6ce2b2f52dc4bd3b04784e00721dbed P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:12.647936 31608 raft_consensus.cc:729] T a6ce2b2f52dc4bd3b04784e00721dbed P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:12:12.648169 31608 consensus_queue.cc:260] T a6ce2b2f52dc4bd3b04784e00721dbed P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:12.648694 31608 ts_tablet_manager.cc:1398] T a6ce2b2f52dc4bd3b04784e00721dbed P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:12.648877 31551 raft_consensus.cc:482] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:12.649011 31551 raft_consensus.cc:504] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:12.650465 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66b8b9c025b941a1a1b6e681fa0bbe90" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:12.650691 30672 raft_consensus.cc:2454] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:12:12.651089 27694 leader_election.cc:302] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:12.651343 31589 raft_consensus.cc:2781] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:12.651428 31589 raft_consensus.cc:482] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:12.651526 31589 raft_consensus.cc:3037] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.652382 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66b8b9c025b941a1a1b6e681fa0bbe90" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:12.652630 30803 raft_consensus.cc:2454] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:12:12.654511 31551 leader_election.cc:288] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:12.655699 31580 tablet_bootstrap.cc:492] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:12.655885 31580 ts_tablet_manager.cc:1367] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.009s
I20240501 08:12:12.655962 31589 raft_consensus.cc:504] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:12.656761 31580 raft_consensus.cc:348] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:12.656912 31589 leader_election.cc:288] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Requested vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:12.657020 31580 raft_consensus.cc:374] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:12.657117 31580 raft_consensus.cc:729] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:12.657343 31580 consensus_queue.cc:260] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:12.658067 31580 ts_tablet_manager.cc:1398] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240501 08:12:12.658471 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66b8b9c025b941a1a1b6e681fa0bbe90" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:12.658723 30804 raft_consensus.cc:3037] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.661659 30804 raft_consensus.cc:2454] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:12:12.662909 27695 leader_election.cc:302] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 64998323a1f4482d8704bf20594d13a5; no voters:
I20240501 08:12:12.663276 31176 raft_consensus.cc:2781] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:12.663453 31176 raft_consensus.cc:686] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:12:12.663772 31176 consensus_queue.cc:237] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:12.664489 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66b8b9c025b941a1a1b6e681fa0bbe90" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:12.664727 30672 raft_consensus.cc:3037] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.666889 30672 raft_consensus.cc:2454] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:12:12.668335 27613 catalog_manager.cc:5484] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:12.715107 31592 raft_consensus.cc:482] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:12.715288 31592 raft_consensus.cc:504] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:12.716104 31592 leader_election.cc:288] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:12.717005 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6ce2b2f52dc4bd3b04784e00721dbed" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:12.717259 30936 raft_consensus.cc:2454] T a6ce2b2f52dc4bd3b04784e00721dbed P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 0.
I20240501 08:12:12.717769 30607 leader_election.cc:302] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a022e1a501f4d44a0468054d6c3de47, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:12.718072 31592 raft_consensus.cc:2781] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:12.718195 31592 raft_consensus.cc:482] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:12.718278 31592 raft_consensus.cc:3037] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.721643 31592 raft_consensus.cc:504] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:12.722836 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6ce2b2f52dc4bd3b04784e00721dbed" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:12.723057 30936 raft_consensus.cc:3037] T a6ce2b2f52dc4bd3b04784e00721dbed P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.723459 31592 leader_election.cc:288] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 election: Requested vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:12.725847 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6ce2b2f52dc4bd3b04784e00721dbed" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:12:12.725924 30936 raft_consensus.cc:2454] T a6ce2b2f52dc4bd3b04784e00721dbed P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 1.
I20240501 08:12:12.726037 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6ce2b2f52dc4bd3b04784e00721dbed" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:12:12.726109 27759 raft_consensus.cc:2454] T a6ce2b2f52dc4bd3b04784e00721dbed P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 0.
I20240501 08:12:12.726459 30607 leader_election.cc:302] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a022e1a501f4d44a0468054d6c3de47, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:12.726732 31592 raft_consensus.cc:2781] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:12.726866 31592 raft_consensus.cc:686] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Becoming Leader. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Running, Role: LEADER
I20240501 08:12:12.727150 31592 consensus_queue.cc:237] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:12.730384 27613 catalog_manager.cc:5484] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134). New cstate: current_term: 1 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:12.836952 27613 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:12.842027 27759 raft_consensus.cc:3037] T a6ce2b2f52dc4bd3b04784e00721dbed P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.842911 30936 raft_consensus.cc:1261] T a6ce2b2f52dc4bd3b04784e00721dbed P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:12.843541 31538 consensus_queue.cc:1035] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:12.845039 27759 raft_consensus.cc:1261] T a6ce2b2f52dc4bd3b04784e00721dbed P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:12.845623 31592 consensus_queue.cc:1035] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:12.849162 27611 catalog_manager.cc:5576] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845) does not have the latest schema for tablet a6ce2b2f52dc4bd3b04784e00721dbed (table rebalancer_dynamic_table [id=cbddc790ca4147c6939b2b1169e03118]). Expected version 1 got 0
I20240501 08:12:12.849433 31593 tablet.cc:1722] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:12.849968 31593 tablet.cc:1620] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:12.856181 31637 tablet.cc:1722] T a6ce2b2f52dc4bd3b04784e00721dbed P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:12.857439 31637 tablet.cc:1620] T a6ce2b2f52dc4bd3b04784e00721dbed P a29cd590999b4f32849ff5689cd53c11: MemRowSet was empty: no flush needed.
I20240501 08:12:12.857898 27613 catalog_manager.cc:5576] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845) does not have the latest schema for tablet a6ce2b2f52dc4bd3b04784e00721dbed (table rebalancer_dynamic_table [id=cbddc790ca4147c6939b2b1169e03118]). Expected version 1 got 0
I20240501 08:12:12.860157 27611 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=cbddc790ca4147c6939b2b1169e03118] alter complete (version 1)
I20240501 08:12:12.866145 27613 catalog_manager.cc:5576] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471) does not have the latest schema for tablet a6ce2b2f52dc4bd3b04784e00721dbed (table rebalancer_dynamic_table [id=cbddc790ca4147c6939b2b1169e03118]). Expected version 1 got 0
I20240501 08:12:12.868589 31639 tablet.cc:1722] T a6ce2b2f52dc4bd3b04784e00721dbed P 6331f77dae9d452184e8cd0cbf5f1b43: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:12.869356 31639 tablet.cc:1620] T a6ce2b2f52dc4bd3b04784e00721dbed P 6331f77dae9d452184e8cd0cbf5f1b43: MemRowSet was empty: no flush needed.
I20240501 08:12:12.870553 27611 catalog_manager.cc:5576] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471) does not have the latest schema for tablet a6ce2b2f52dc4bd3b04784e00721dbed (table rebalancer_dynamic_table [id=cbddc790ca4147c6939b2b1169e03118]). Expected version 1 got 0
I20240501 08:12:12.914131 28150 raft_consensus.cc:3037] T 98525304f2334bf0a3f52c06df58b3ba P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:12.916710 31616 consensus_queue.cc:1035] T 98525304f2334bf0a3f52c06df58b3ba P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:12.921170 31616 consensus_queue.cc:1035] T 98525304f2334bf0a3f52c06df58b3ba P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:12.965055 27613 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:12.972414 31593 tablet.cc:1722] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:12.973062 31593 tablet.cc:1620] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:12.973302 31639 tablet.cc:1722] T a6ce2b2f52dc4bd3b04784e00721dbed P 6331f77dae9d452184e8cd0cbf5f1b43: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:12.973452 31637 tablet.cc:1722] T a6ce2b2f52dc4bd3b04784e00721dbed P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:12.973891 31639 tablet.cc:1620] T a6ce2b2f52dc4bd3b04784e00721dbed P 6331f77dae9d452184e8cd0cbf5f1b43: MemRowSet was empty: no flush needed.
I20240501 08:12:12.973970 31637 tablet.cc:1620] T a6ce2b2f52dc4bd3b04784e00721dbed P a29cd590999b4f32849ff5689cd53c11: MemRowSet was empty: no flush needed.
I20240501 08:12:12.982321 27613 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=cbddc790ca4147c6939b2b1169e03118] alter complete (version 2)
I20240501 08:12:13.020560 31599 consensus_queue.cc:1035] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:13.025254 30672 raft_consensus.cc:3037] T d6bb70263134414b9945a87087b55a2c P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:13.028784 31647 consensus_queue.cc:1035] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:13.085815 27611 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:13.093227 31593 tablet.cc:1722] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:13.100091 31637 tablet.cc:1722] T a6ce2b2f52dc4bd3b04784e00721dbed P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:13.102985 31639 tablet.cc:1722] T a6ce2b2f52dc4bd3b04784e00721dbed P 6331f77dae9d452184e8cd0cbf5f1b43: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:13.111322 27611 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table_new_name [id=cbddc790ca4147c6939b2b1169e03118] alter complete (version 3)
I20240501 08:12:13.136436 31628 consensus_queue.cc:1035] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:13.142990 31628 consensus_queue.cc:1035] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:13.220613 27611 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:13.220851 27611 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:13.224296 27611 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet a6ce2b2f52dc4bd3b04784e00721dbed
I20240501 08:12:13.225155 30917 tablet_service.cc:1514] Processing DeleteTablet for tablet a6ce2b2f52dc4bd3b04784e00721dbed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:13 UTC) from {username='slave'} at 127.0.0.1:54036
I20240501 08:12:13.225648 30651 tablet_service.cc:1514] Processing DeleteTablet for tablet a6ce2b2f52dc4bd3b04784e00721dbed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:13 UTC) from {username='slave'} at 127.0.0.1:57156
I20240501 08:12:13.226095 31653 tablet_replica.cc:330] T a6ce2b2f52dc4bd3b04784e00721dbed P a29cd590999b4f32849ff5689cd53c11: stopping tablet replica
I20240501 08:12:13.226310 31653 raft_consensus.cc:2229] T a6ce2b2f52dc4bd3b04784e00721dbed P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:13.226796 27741 tablet_service.cc:1514] Processing DeleteTablet for tablet a6ce2b2f52dc4bd3b04784e00721dbed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:13 UTC) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:13.226996 31654 tablet_replica.cc:330] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47: stopping tablet replica
I20240501 08:12:13.227206 31654 raft_consensus.cc:2229] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:13.226600 31653 raft_consensus.cc:2258] T a6ce2b2f52dc4bd3b04784e00721dbed P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:13.227535 31655 tablet_replica.cc:330] T a6ce2b2f52dc4bd3b04784e00721dbed P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:13.227627 31654 raft_consensus.cc:2258] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:13.227775 31655 raft_consensus.cc:2229] T a6ce2b2f52dc4bd3b04784e00721dbed P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:13.228041 31655 raft_consensus.cc:2258] T a6ce2b2f52dc4bd3b04784e00721dbed P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:13.228368 31653 ts_tablet_manager.cc:1875] T a6ce2b2f52dc4bd3b04784e00721dbed P a29cd590999b4f32849ff5689cd53c11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:13.228480 31654 ts_tablet_manager.cc:1875] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:13.228760 31655 ts_tablet_manager.cc:1875] T a6ce2b2f52dc4bd3b04784e00721dbed P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:13.234992 31655 ts_tablet_manager.cc:1888] T a6ce2b2f52dc4bd3b04784e00721dbed P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:13.234999 31653 ts_tablet_manager.cc:1888] T a6ce2b2f52dc4bd3b04784e00721dbed P a29cd590999b4f32849ff5689cd53c11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:13.235159 31653 log.cc:1196] T a6ce2b2f52dc4bd3b04784e00721dbed P a29cd590999b4f32849ff5689cd53c11: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/wals/a6ce2b2f52dc4bd3b04784e00721dbed
I20240501 08:12:13.235216 31655 log.cc:1196] T a6ce2b2f52dc4bd3b04784e00721dbed P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/a6ce2b2f52dc4bd3b04784e00721dbed
I20240501 08:12:13.235251 31654 ts_tablet_manager.cc:1888] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:13.235409 31654 log.cc:1196] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/wals/a6ce2b2f52dc4bd3b04784e00721dbed
I20240501 08:12:13.235734 31655 ts_tablet_manager.cc:1909] T a6ce2b2f52dc4bd3b04784e00721dbed P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting consensus metadata
I20240501 08:12:13.235734 31653 ts_tablet_manager.cc:1909] T a6ce2b2f52dc4bd3b04784e00721dbed P a29cd590999b4f32849ff5689cd53c11: Deleting consensus metadata
I20240501 08:12:13.235846 31654 ts_tablet_manager.cc:1909] T a6ce2b2f52dc4bd3b04784e00721dbed P 9a022e1a501f4d44a0468054d6c3de47: Deleting consensus metadata
I20240501 08:12:13.236940 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet a6ce2b2f52dc4bd3b04784e00721dbed (table rebalancer_dynamic_table_new_name [id=cbddc790ca4147c6939b2b1169e03118]) successfully deleted
I20240501 08:12:13.237228 27606 catalog_manager.cc:4830] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): tablet a6ce2b2f52dc4bd3b04784e00721dbed (table rebalancer_dynamic_table_new_name [id=cbddc790ca4147c6939b2b1169e03118]) successfully deleted
I20240501 08:12:13.237406 27604 catalog_manager.cc:4830] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): tablet a6ce2b2f52dc4bd3b04784e00721dbed (table rebalancer_dynamic_table_new_name [id=cbddc790ca4147c6939b2b1169e03118]) successfully deleted
I20240501 08:12:13.256956 27611 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_extra_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:13.267012 30917 tablet_service.cc:1467] Processing CreateTablet for tablet 98c8e3a6f66a42b08507668f7c3a8af1 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=c906ec3019644e00bd7c63c8bcc3d323]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:13.267129 27872 tablet_service.cc:1467] Processing CreateTablet for tablet 98c8e3a6f66a42b08507668f7c3a8af1 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=c906ec3019644e00bd7c63c8bcc3d323]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:13.267520 30917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98c8e3a6f66a42b08507668f7c3a8af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:13.267578 27872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98c8e3a6f66a42b08507668f7c3a8af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:13.267560 30517 tablet_service.cc:1467] Processing CreateTablet for tablet 7c3b2b31a9e747c69ff67969d878bc64 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=c906ec3019644e00bd7c63c8bcc3d323]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:13.267977 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c3b2b31a9e747c69ff67969d878bc64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:13.269089 30514 tablet_service.cc:1467] Processing CreateTablet for tablet b6b0bcb4d07e4125bccf1711a094576e (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=c906ec3019644e00bd7c63c8bcc3d323]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:13.269526 30514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6b0bcb4d07e4125bccf1711a094576e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:13.274382 30915 tablet_service.cc:1467] Processing CreateTablet for tablet 7c3b2b31a9e747c69ff67969d878bc64 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=c906ec3019644e00bd7c63c8bcc3d323]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:13.274843 30915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c3b2b31a9e747c69ff67969d878bc64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:13.277190 30651 tablet_service.cc:1467] Processing CreateTablet for tablet b6b0bcb4d07e4125bccf1711a094576e (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=c906ec3019644e00bd7c63c8bcc3d323]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:13.277617 30651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6b0bcb4d07e4125bccf1711a094576e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:13.278718 30916 tablet_service.cc:1467] Processing CreateTablet for tablet b6b0bcb4d07e4125bccf1711a094576e (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=c906ec3019644e00bd7c63c8bcc3d323]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:13.278877 31656 tablet_bootstrap.cc:492] T b6b0bcb4d07e4125bccf1711a094576e P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:13.279115 30916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6b0bcb4d07e4125bccf1711a094576e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:13.280484 31656 tablet_bootstrap.cc:654] T b6b0bcb4d07e4125bccf1711a094576e P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:13.282837 28131 tablet_service.cc:1467] Processing CreateTablet for tablet 7c3b2b31a9e747c69ff67969d878bc64 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=c906ec3019644e00bd7c63c8bcc3d323]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:13.283293 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c3b2b31a9e747c69ff67969d878bc64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:13.282838 28132 tablet_service.cc:1467] Processing CreateTablet for tablet 98c8e3a6f66a42b08507668f7c3a8af1 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=c906ec3019644e00bd7c63c8bcc3d323]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:13.283746 28132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98c8e3a6f66a42b08507668f7c3a8af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:13.293418 31656 tablet_bootstrap.cc:492] T b6b0bcb4d07e4125bccf1711a094576e P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:13.293664 31656 ts_tablet_manager.cc:1367] T b6b0bcb4d07e4125bccf1711a094576e P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240501 08:12:13.294251 31658 tablet_bootstrap.cc:492] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:13.294991 31656 raft_consensus.cc:348] T b6b0bcb4d07e4125bccf1711a094576e P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:13.295261 31656 raft_consensus.cc:374] T b6b0bcb4d07e4125bccf1711a094576e P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:13.295392 31656 raft_consensus.cc:729] T b6b0bcb4d07e4125bccf1711a094576e P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:13.296957 31658 tablet_bootstrap.cc:654] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:13.297735 31656 consensus_queue.cc:260] T b6b0bcb4d07e4125bccf1711a094576e P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:13.302646 31656 ts_tablet_manager.cc:1398] T b6b0bcb4d07e4125bccf1711a094576e P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240501 08:12:13.303030 31656 tablet_bootstrap.cc:492] T 7c3b2b31a9e747c69ff67969d878bc64 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:13.310253 31656 tablet_bootstrap.cc:654] T 7c3b2b31a9e747c69ff67969d878bc64 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:13.313347 31662 tablet_bootstrap.cc:492] T b6b0bcb4d07e4125bccf1711a094576e P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:13.314443 31659 tablet_bootstrap.cc:492] T 98c8e3a6f66a42b08507668f7c3a8af1 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:13.316506 31659 tablet_bootstrap.cc:654] T 98c8e3a6f66a42b08507668f7c3a8af1 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:13.317157 31656 tablet_bootstrap.cc:492] T 7c3b2b31a9e747c69ff67969d878bc64 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:13.317335 31656 ts_tablet_manager.cc:1367] T 7c3b2b31a9e747c69ff67969d878bc64 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.001s
I20240501 08:12:13.318248 31656 raft_consensus.cc:348] T 7c3b2b31a9e747c69ff67969d878bc64 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:13.318499 31656 raft_consensus.cc:374] T 7c3b2b31a9e747c69ff67969d878bc64 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:13.318583 31656 raft_consensus.cc:729] T 7c3b2b31a9e747c69ff67969d878bc64 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:13.318601 31658 tablet_bootstrap.cc:492] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:13.318763 31658 ts_tablet_manager.cc:1367] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.012s
I20240501 08:12:13.319085 31659 tablet_bootstrap.cc:492] T 98c8e3a6f66a42b08507668f7c3a8af1 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:12:13.319568 31659 ts_tablet_manager.cc:1367] T 98c8e3a6f66a42b08507668f7c3a8af1 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240501 08:12:13.319499 31658 raft_consensus.cc:348] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:13.319662 31658 raft_consensus.cc:374] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:13.319767 31658 raft_consensus.cc:729] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:13.319998 31658 consensus_queue.cc:260] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:13.320363 31659 raft_consensus.cc:348] T 98c8e3a6f66a42b08507668f7c3a8af1 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:13.320523 31662 tablet_bootstrap.cc:654] T b6b0bcb4d07e4125bccf1711a094576e P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:13.320641 31658 ts_tablet_manager.cc:1398] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:13.320664 31659 raft_consensus.cc:374] T 98c8e3a6f66a42b08507668f7c3a8af1 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:13.320760 31659 raft_consensus.cc:729] T 98c8e3a6f66a42b08507668f7c3a8af1 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:12:13.321033 31658 tablet_bootstrap.cc:492] T 7c3b2b31a9e747c69ff67969d878bc64 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:13.321084 31659 consensus_queue.cc:260] T 98c8e3a6f66a42b08507668f7c3a8af1 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:13.321710 31659 ts_tablet_manager.cc:1398] T 98c8e3a6f66a42b08507668f7c3a8af1 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:13.324295 31658 tablet_bootstrap.cc:654] T 7c3b2b31a9e747c69ff67969d878bc64 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:13.324950 31660 tablet_bootstrap.cc:492] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:13.326694 31656 consensus_queue.cc:260] T 7c3b2b31a9e747c69ff67969d878bc64 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:13.327793 27617 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:13.328322 31656 ts_tablet_manager.cc:1398] T 7c3b2b31a9e747c69ff67969d878bc64 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 08:12:13.329898 31660 tablet_bootstrap.cc:654] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:13.334053 31658 tablet_bootstrap.cc:492] T 7c3b2b31a9e747c69ff67969d878bc64 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:13.334254 31658 ts_tablet_manager.cc:1367] T 7c3b2b31a9e747c69ff67969d878bc64 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.002s
I20240501 08:12:13.335073 31662 tablet_bootstrap.cc:492] T b6b0bcb4d07e4125bccf1711a094576e P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:13.335161 31658 raft_consensus.cc:348] T 7c3b2b31a9e747c69ff67969d878bc64 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:13.335393 31658 raft_consensus.cc:374] T 7c3b2b31a9e747c69ff67969d878bc64 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:13.335440 31662 ts_tablet_manager.cc:1367] T b6b0bcb4d07e4125bccf1711a094576e P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20240501 08:12:13.335480 31658 raft_consensus.cc:729] T 7c3b2b31a9e747c69ff67969d878bc64 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:13.335780 31658 consensus_queue.cc:260] T 7c3b2b31a9e747c69ff67969d878bc64 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:13.336409 31658 ts_tablet_manager.cc:1398] T 7c3b2b31a9e747c69ff67969d878bc64 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:13.336560 30651 tablet_service.cc:1467] Processing CreateTablet for tablet cfef339b34e8403bb91e17a2fc400834 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f5fae2fca0e04b36af07b0519bdc94c0]), partition=
I20240501 08:12:13.336974 30651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfef339b34e8403bb91e17a2fc400834. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:13.338376 28002 tablet_service.cc:1467] Processing CreateTablet for tablet cfef339b34e8403bb91e17a2fc400834 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f5fae2fca0e04b36af07b0519bdc94c0]), partition=
I20240501 08:12:13.338766 28002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfef339b34e8403bb91e17a2fc400834. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:13.339586 31658 tablet_bootstrap.cc:492] T 98c8e3a6f66a42b08507668f7c3a8af1 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:13.341308 31658 tablet_bootstrap.cc:654] T 98c8e3a6f66a42b08507668f7c3a8af1 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:13.341595 30917 tablet_service.cc:1467] Processing CreateTablet for tablet cfef339b34e8403bb91e17a2fc400834 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f5fae2fca0e04b36af07b0519bdc94c0]), partition=
I20240501 08:12:13.341969 30917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfef339b34e8403bb91e17a2fc400834. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:13.344421 31658 tablet_bootstrap.cc:492] T 98c8e3a6f66a42b08507668f7c3a8af1 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:13.344573 31658 ts_tablet_manager.cc:1367] T 98c8e3a6f66a42b08507668f7c3a8af1 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20240501 08:12:13.345412 31658 raft_consensus.cc:348] T 98c8e3a6f66a42b08507668f7c3a8af1 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:13.346319 31658 raft_consensus.cc:374] T 98c8e3a6f66a42b08507668f7c3a8af1 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:13.346428 31658 raft_consensus.cc:729] T 98c8e3a6f66a42b08507668f7c3a8af1 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:13.346688 31658 consensus_queue.cc:260] T 98c8e3a6f66a42b08507668f7c3a8af1 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:13.347254 31658 ts_tablet_manager.cc:1398] T 98c8e3a6f66a42b08507668f7c3a8af1 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:12:13.352888 31669 tablet_bootstrap.cc:492] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:13.354979 31658 tablet_bootstrap.cc:492] T cfef339b34e8403bb91e17a2fc400834 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:13.355182 31669 tablet_bootstrap.cc:654] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:13.356232 31662 raft_consensus.cc:348] T b6b0bcb4d07e4125bccf1711a094576e P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:13.356520 31662 raft_consensus.cc:374] T b6b0bcb4d07e4125bccf1711a094576e P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:13.356632 31662 raft_consensus.cc:729] T b6b0bcb4d07e4125bccf1711a094576e P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:13.356724 31658 tablet_bootstrap.cc:654] T cfef339b34e8403bb91e17a2fc400834 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:13.356897 31662 consensus_queue.cc:260] T b6b0bcb4d07e4125bccf1711a094576e P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:13.357522 31662 ts_tablet_manager.cc:1398] T b6b0bcb4d07e4125bccf1711a094576e P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240501 08:12:13.357995 31662 tablet_bootstrap.cc:492] T cfef339b34e8403bb91e17a2fc400834 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:13.360702 31660 tablet_bootstrap.cc:492] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:12:13.361156 31660 ts_tablet_manager.cc:1367] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.003s
I20240501 08:12:13.362084 31660 raft_consensus.cc:348] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:13.362345 31662 tablet_bootstrap.cc:654] T cfef339b34e8403bb91e17a2fc400834 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:13.363328 31660 raft_consensus.cc:374] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:13.363487 31660 raft_consensus.cc:729] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:12:13.363829 31660 consensus_queue.cc:260] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:13.364444 31660 ts_tablet_manager.cc:1398] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:13.364826 31660 tablet_bootstrap.cc:492] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:13.365711 31615 raft_consensus.cc:482] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:13.365965 31615 raft_consensus.cc:504] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:13.366873 31615 leader_election.cc:288] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:13.367733 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b0bcb4d07e4125bccf1711a094576e" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:13.368038 30536 raft_consensus.cc:2454] T b6b0bcb4d07e4125bccf1711a094576e P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:13.368588 30873 leader_election.cc:302] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:13.369369 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b0bcb4d07e4125bccf1711a094576e" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:13.369680 30672 raft_consensus.cc:2454] T b6b0bcb4d07e4125bccf1711a094576e P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:13.370564 31615 raft_consensus.cc:2781] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:13.370716 31615 raft_consensus.cc:482] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:13.370855 31615 raft_consensus.cc:3037] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:13.371730 31669 tablet_bootstrap.cc:492] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:12:13.372009 31669 ts_tablet_manager.cc:1367] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240501 08:12:13.373131 31669 raft_consensus.cc:348] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:13.373816 31669 raft_consensus.cc:374] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:13.382051 31669 raft_consensus.cc:729] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:12:13.382619 31669 consensus_queue.cc:260] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:13.384104 31660 tablet_bootstrap.cc:654] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:13.385921 31615 raft_consensus.cc:504] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:13.387284 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b0bcb4d07e4125bccf1711a094576e" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:13.387513 30537 raft_consensus.cc:3037] T b6b0bcb4d07e4125bccf1711a094576e P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:13.390954 30537 raft_consensus.cc:2454] T b6b0bcb4d07e4125bccf1711a094576e P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:13.391074 31669 ts_tablet_manager.cc:1398] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.019s user 0.003s sys 0.000s
I20240501 08:12:13.391633 30873 leader_election.cc:302] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:13.391975 31616 raft_consensus.cc:2781] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:13.392125 31616 raft_consensus.cc:686] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Becoming Leader. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:12:13.392410 31616 consensus_queue.cc:237] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:13.396972 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b0bcb4d07e4125bccf1711a094576e" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:13.397264 30672 raft_consensus.cc:3037] T b6b0bcb4d07e4125bccf1711a094576e P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:13.402245 30672 raft_consensus.cc:2454] T b6b0bcb4d07e4125bccf1711a094576e P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:13.404673 31615 leader_election.cc:288] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Requested vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:13.415338 31658 tablet_bootstrap.cc:492] T cfef339b34e8403bb91e17a2fc400834 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:13.415598 31658 ts_tablet_manager.cc:1367] T cfef339b34e8403bb91e17a2fc400834 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.061s user 0.028s sys 0.025s
I20240501 08:12:13.416641 31658 raft_consensus.cc:348] T cfef339b34e8403bb91e17a2fc400834 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:13.416957 31658 raft_consensus.cc:374] T cfef339b34e8403bb91e17a2fc400834 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:13.417053 31658 raft_consensus.cc:729] T cfef339b34e8403bb91e17a2fc400834 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:13.420517 31658 consensus_queue.cc:260] T cfef339b34e8403bb91e17a2fc400834 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:13.421308 31658 ts_tablet_manager.cc:1398] T cfef339b34e8403bb91e17a2fc400834 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20240501 08:12:13.421666 31660 tablet_bootstrap.cc:492] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:12:13.421906 31660 ts_tablet_manager.cc:1367] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.057s user 0.017s sys 0.010s
I20240501 08:12:13.422902 31660 raft_consensus.cc:348] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:13.422863 27617 catalog_manager.cc:5484] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 reported cstate change: term changed from 0 to 1, leader changed from <none> to a29cd590999b4f32849ff5689cd53c11 (127.21.84.136). New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:13.423318 31660 raft_consensus.cc:374] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:13.423447 31660 raft_consensus.cc:729] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:12:13.425237 31662 tablet_bootstrap.cc:492] T cfef339b34e8403bb91e17a2fc400834 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:13.425419 31662 ts_tablet_manager.cc:1367] T cfef339b34e8403bb91e17a2fc400834 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.068s user 0.004s sys 0.021s
I20240501 08:12:13.423851 31660 consensus_queue.cc:260] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:13.429548 31662 raft_consensus.cc:348] T cfef339b34e8403bb91e17a2fc400834 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:13.429841 31662 raft_consensus.cc:374] T cfef339b34e8403bb91e17a2fc400834 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:13.430332 31662 raft_consensus.cc:729] T cfef339b34e8403bb91e17a2fc400834 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:13.430951 31662 consensus_queue.cc:260] T cfef339b34e8403bb91e17a2fc400834 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:13.431934 31660 ts_tablet_manager.cc:1398] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240501 08:12:13.435951 31662 ts_tablet_manager.cc:1398] T cfef339b34e8403bb91e17a2fc400834 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.010s user 0.000s sys 0.003s
I20240501 08:12:13.458827 31251 raft_consensus.cc:482] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:13.459003 31251 raft_consensus.cc:504] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:13.459798 31251 leader_election.cc:288] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:12:13.460461 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c8e3a6f66a42b08507668f7c3a8af1" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:12:13.460775 27890 raft_consensus.cc:2454] T 98c8e3a6f66a42b08507668f7c3a8af1 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:12:13.461275 28088 leader_election.cc:302] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:13.461606 31251 raft_consensus.cc:2781] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:13.461750 31251 raft_consensus.cc:482] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:13.461865 31251 raft_consensus.cc:3037] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:13.464200 31251 raft_consensus.cc:504] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:13.464979 31251 leader_election.cc:288] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:12:13.465461 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c8e3a6f66a42b08507668f7c3a8af1" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:12:13.465670 27890 raft_consensus.cc:3037] T 98c8e3a6f66a42b08507668f7c3a8af1 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:13.467494 30937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c8e3a6f66a42b08507668f7c3a8af1" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:13.467679 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c8e3a6f66a42b08507668f7c3a8af1" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:13.467739 30937 raft_consensus.cc:2454] T 98c8e3a6f66a42b08507668f7c3a8af1 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:12:13.467868 30936 raft_consensus.cc:3037] T 98c8e3a6f66a42b08507668f7c3a8af1 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:13.468238 27890 raft_consensus.cc:2454] T 98c8e3a6f66a42b08507668f7c3a8af1 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:12:13.469874 30936 raft_consensus.cc:2454] T 98c8e3a6f66a42b08507668f7c3a8af1 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:12:13.470119 28088 leader_election.cc:302] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:13.470501 31251 raft_consensus.cc:2781] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:13.470669 31251 raft_consensus.cc:686] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:12:13.470953 31251 consensus_queue.cc:237] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:13.477284 27613 catalog_manager.cc:5484] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:13.493178 31251 raft_consensus.cc:482] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:13.493422 31251 raft_consensus.cc:504] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:13.494071 31251 leader_election.cc:288] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:13.494473 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c3b2b31a9e747c69ff67969d878bc64" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:13.494560 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c3b2b31a9e747c69ff67969d878bc64" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:13.494717 30537 raft_consensus.cc:2454] T 7c3b2b31a9e747c69ff67969d878bc64 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:12:13.494778 30936 raft_consensus.cc:2454] T 7c3b2b31a9e747c69ff67969d878bc64 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:12:13.495126 28088 leader_election.cc:302] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:12:13.495397 31251 raft_consensus.cc:2781] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:13.495496 31251 raft_consensus.cc:482] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:13.495608 31251 raft_consensus.cc:3037] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:13.497762 31251 raft_consensus.cc:504] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:13.498395 31251 leader_election.cc:288] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:13.498864 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c3b2b31a9e747c69ff67969d878bc64" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:13.499119 30537 raft_consensus.cc:3037] T 7c3b2b31a9e747c69ff67969d878bc64 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:13.499203 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c3b2b31a9e747c69ff67969d878bc64" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:13.499495 30936 raft_consensus.cc:3037] T 7c3b2b31a9e747c69ff67969d878bc64 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:13.502084 30537 raft_consensus.cc:2454] T 7c3b2b31a9e747c69ff67969d878bc64 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:12:13.502295 30936 raft_consensus.cc:2454] T 7c3b2b31a9e747c69ff67969d878bc64 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:12:13.502499 28088 leader_election.cc:302] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:12:13.502784 31251 raft_consensus.cc:2781] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:13.503001 31251 raft_consensus.cc:686] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:12:13.503273 31251 consensus_queue.cc:237] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:13.506474 27613 catalog_manager.cc:5484] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:13.699626 31652 raft_consensus.cc:482] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:13.699896 31652 raft_consensus.cc:504] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:13.700888 31652 leader_election.cc:288] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:13.715101 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfef339b34e8403bb91e17a2fc400834" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:13.715387 30936 raft_consensus.cc:2454] T cfef339b34e8403bb91e17a2fc400834 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 0.
I20240501 08:12:13.715824 27957 leader_election.cc:302] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a29cd590999b4f32849ff5689cd53c11, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:12:13.716150 31652 raft_consensus.cc:2781] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:13.716270 31652 raft_consensus.cc:482] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:13.716347 31652 raft_consensus.cc:3037] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:13.716627 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfef339b34e8403bb91e17a2fc400834" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:13.716805 30672 raft_consensus.cc:2454] T cfef339b34e8403bb91e17a2fc400834 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 0.
I20240501 08:12:13.721592 31652 raft_consensus.cc:504] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:13.722801 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfef339b34e8403bb91e17a2fc400834" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:13.723032 30936 raft_consensus.cc:3037] T cfef339b34e8403bb91e17a2fc400834 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:13.723057 31652 leader_election.cc:288] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Requested vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:13.723961 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfef339b34e8403bb91e17a2fc400834" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:13.724161 30672 raft_consensus.cc:3037] T cfef339b34e8403bb91e17a2fc400834 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:13.725116 30936 raft_consensus.cc:2454] T cfef339b34e8403bb91e17a2fc400834 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 1.
I20240501 08:12:13.725616 27957 leader_election.cc:302] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a29cd590999b4f32849ff5689cd53c11, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:12:13.725970 31652 raft_consensus.cc:2781] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:13.726125 31652 raft_consensus.cc:686] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Becoming Leader. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Running, Role: LEADER
I20240501 08:12:13.726415 31652 consensus_queue.cc:237] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:13.726946 30672 raft_consensus.cc:2454] T cfef339b34e8403bb91e17a2fc400834 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 1.
I20240501 08:12:13.730605 27613 catalog_manager.cc:5484] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to aad81047009a4ffcbc450b60030583b5 (127.21.84.131). New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:13.792361 31616 consensus_queue.cc:1035] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:13.797716 31616 consensus_queue.cc:1035] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:13.910450 27617 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:13.931167 30672 raft_consensus.cc:1261] T cfef339b34e8403bb91e17a2fc400834 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:13.931449 30936 raft_consensus.cc:1261] T cfef339b34e8403bb91e17a2fc400834 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:13.931860 31652 consensus_queue.cc:1035] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:13.932142 31503 consensus_queue.cc:1035] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:13.945348 31499 consensus_queue.cc:1035] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 08:12:13.948493 27617 catalog_manager.cc:5576] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845) does not have the latest schema for tablet cfef339b34e8403bb91e17a2fc400834 (table rebalancer_dynamic_table [id=f5fae2fca0e04b36af07b0519bdc94c0]). Expected version 1 got 0
I20240501 08:12:13.949117 27613 catalog_manager.cc:5576] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933) does not have the latest schema for tablet cfef339b34e8403bb91e17a2fc400834 (table rebalancer_dynamic_table [id=f5fae2fca0e04b36af07b0519bdc94c0]). Expected version 1 got 0
I20240501 08:12:13.958205 31499 consensus_queue.cc:1035] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:13.961664 31693 tablet.cc:1722] T cfef339b34e8403bb91e17a2fc400834 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:13.963848 31693 tablet.cc:1620] T cfef339b34e8403bb91e17a2fc400834 P a29cd590999b4f32849ff5689cd53c11: MemRowSet was empty: no flush needed.
I20240501 08:12:13.964175 27613 catalog_manager.cc:5576] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845) does not have the latest schema for tablet cfef339b34e8403bb91e17a2fc400834 (table rebalancer_dynamic_table [id=f5fae2fca0e04b36af07b0519bdc94c0]). Expected version 1 got 0
I20240501 08:12:13.968703 31691 tablet.cc:1722] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:13.970078 31691 tablet.cc:1620] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5: MemRowSet was empty: no flush needed.
I20240501 08:12:13.972549 31697 consensus_queue.cc:1035] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:13.975879 31697 consensus_queue.cc:1035] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:13.991281 31705 tablet.cc:1722] T cfef339b34e8403bb91e17a2fc400834 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:13.992630 31705 tablet.cc:1620] T cfef339b34e8403bb91e17a2fc400834 P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:13.993355 27617 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=f5fae2fca0e04b36af07b0519bdc94c0] alter complete (version 1)
I20240501 08:12:13.993863 27611 catalog_manager.cc:5576] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933) does not have the latest schema for tablet cfef339b34e8403bb91e17a2fc400834 (table rebalancer_dynamic_table [id=f5fae2fca0e04b36af07b0519bdc94c0]). Expected version 1 got 0
I20240501 08:12:14.051549 27617 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_extra_table_20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:14.063902 27741 tablet_service.cc:1467] Processing CreateTablet for tablet 22cdd207ff0c49e0b89d68de068d3603 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=7884fe25785d4c7e9d8b2559f8e81777]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:14.063954 30651 tablet_service.cc:1467] Processing CreateTablet for tablet 22cdd207ff0c49e0b89d68de068d3603 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=7884fe25785d4c7e9d8b2559f8e81777]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:14.064399 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22cdd207ff0c49e0b89d68de068d3603. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:14.064435 30651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22cdd207ff0c49e0b89d68de068d3603. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:14.068161 27872 tablet_service.cc:1467] Processing CreateTablet for tablet fea1af7c0ecc4ddba679a981125d957a (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=7884fe25785d4c7e9d8b2559f8e81777]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:14.068135 30652 tablet_service.cc:1467] Processing CreateTablet for tablet fea1af7c0ecc4ddba679a981125d957a (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=7884fe25785d4c7e9d8b2559f8e81777]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:14.068570 27872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fea1af7c0ecc4ddba679a981125d957a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:14.068595 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fea1af7c0ecc4ddba679a981125d957a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:14.068480 30917 tablet_service.cc:1467] Processing CreateTablet for tablet 22cdd207ff0c49e0b89d68de068d3603 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=7884fe25785d4c7e9d8b2559f8e81777]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:14.068965 30917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22cdd207ff0c49e0b89d68de068d3603. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:14.073594 30514 tablet_service.cc:1467] Processing CreateTablet for tablet c5f4c78e69fb4c67ad65a7b0c1466798 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=7884fe25785d4c7e9d8b2559f8e81777]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:14.074121 30514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5f4c78e69fb4c67ad65a7b0c1466798. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:14.075471 30784 tablet_service.cc:1467] Processing CreateTablet for tablet c5f4c78e69fb4c67ad65a7b0c1466798 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=7884fe25785d4c7e9d8b2559f8e81777]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:14.075891 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5f4c78e69fb4c67ad65a7b0c1466798. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:14.078305 28002 tablet_service.cc:1467] Processing CreateTablet for tablet c5f4c78e69fb4c67ad65a7b0c1466798 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=7884fe25785d4c7e9d8b2559f8e81777]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:14.078889 28002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5f4c78e69fb4c67ad65a7b0c1466798. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:14.082370 30782 tablet_service.cc:1467] Processing CreateTablet for tablet fea1af7c0ecc4ddba679a981125d957a (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=7884fe25785d4c7e9d8b2559f8e81777]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:14.082803 30782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fea1af7c0ecc4ddba679a981125d957a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:14.092521 31706 tablet_bootstrap.cc:492] T fea1af7c0ecc4ddba679a981125d957a P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:14.096136 31706 tablet_bootstrap.cc:654] T fea1af7c0ecc4ddba679a981125d957a P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:14.099207 27617 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:14.103799 31710 tablet_bootstrap.cc:492] T c5f4c78e69fb4c67ad65a7b0c1466798 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:14.107795 31706 tablet_bootstrap.cc:492] T fea1af7c0ecc4ddba679a981125d957a P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:14.108563 31706 ts_tablet_manager.cc:1367] T fea1af7c0ecc4ddba679a981125d957a P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.003s
I20240501 08:12:14.110805 31706 raft_consensus.cc:348] T fea1af7c0ecc4ddba679a981125d957a P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:14.110960 31693 tablet.cc:1722] T cfef339b34e8403bb91e17a2fc400834 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:14.111407 31693 tablet.cc:1620] T cfef339b34e8403bb91e17a2fc400834 P a29cd590999b4f32849ff5689cd53c11: MemRowSet was empty: no flush needed.
I20240501 08:12:14.111616 31706 raft_consensus.cc:374] T fea1af7c0ecc4ddba679a981125d957a P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:14.111759 31691 tablet.cc:1722] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:14.111891 31706 raft_consensus.cc:729] T fea1af7c0ecc4ddba679a981125d957a P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:14.112296 31691 tablet.cc:1620] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5: MemRowSet was empty: no flush needed.
I20240501 08:12:14.112335 31706 consensus_queue.cc:260] T fea1af7c0ecc4ddba679a981125d957a P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:14.114789 31707 tablet_bootstrap.cc:492] T 22cdd207ff0c49e0b89d68de068d3603 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:14.115424 31705 tablet.cc:1722] T cfef339b34e8403bb91e17a2fc400834 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:14.115892 31705 tablet.cc:1620] T cfef339b34e8403bb91e17a2fc400834 P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:14.116638 31713 tablet_bootstrap.cc:492] T 22cdd207ff0c49e0b89d68de068d3603 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:14.118170 31707 tablet_bootstrap.cc:654] T 22cdd207ff0c49e0b89d68de068d3603 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:14.122300 31706 ts_tablet_manager.cc:1398] T fea1af7c0ecc4ddba679a981125d957a P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240501 08:12:14.123539 27617 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=f5fae2fca0e04b36af07b0519bdc94c0] alter complete (version 2)
I20240501 08:12:14.126765 31713 tablet_bootstrap.cc:654] T 22cdd207ff0c49e0b89d68de068d3603 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:14.133962 31706 tablet_bootstrap.cc:492] T 22cdd207ff0c49e0b89d68de068d3603 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:14.136268 31706 tablet_bootstrap.cc:654] T 22cdd207ff0c49e0b89d68de068d3603 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:14.137322 31708 tablet_bootstrap.cc:492] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:14.139631 31708 tablet_bootstrap.cc:654] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:14.144671 31710 tablet_bootstrap.cc:654] T c5f4c78e69fb4c67ad65a7b0c1466798 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:14.151039 31711 tablet_bootstrap.cc:492] T fea1af7c0ecc4ddba679a981125d957a P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:14.152328 31711 tablet_bootstrap.cc:654] T fea1af7c0ecc4ddba679a981125d957a P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:14.152598 31712 tablet_bootstrap.cc:492] T c5f4c78e69fb4c67ad65a7b0c1466798 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:14.155407 31712 tablet_bootstrap.cc:654] T c5f4c78e69fb4c67ad65a7b0c1466798 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:14.156360 31707 tablet_bootstrap.cc:492] T 22cdd207ff0c49e0b89d68de068d3603 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:14.156538 31707 ts_tablet_manager.cc:1367] T 22cdd207ff0c49e0b89d68de068d3603 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.009s
I20240501 08:12:14.157161 31708 tablet_bootstrap.cc:492] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:14.157670 31707 raft_consensus.cc:348] T 22cdd207ff0c49e0b89d68de068d3603 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:14.158002 31707 raft_consensus.cc:374] T 22cdd207ff0c49e0b89d68de068d3603 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:14.158120 31707 raft_consensus.cc:729] T 22cdd207ff0c49e0b89d68de068d3603 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:14.158469 31710 tablet_bootstrap.cc:492] T c5f4c78e69fb4c67ad65a7b0c1466798 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:14.158416 31707 consensus_queue.cc:260] T 22cdd207ff0c49e0b89d68de068d3603 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:14.158792 31708 ts_tablet_manager.cc:1367] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.056s user 0.008s sys 0.000s
I20240501 08:12:14.158949 31710 ts_tablet_manager.cc:1367] T c5f4c78e69fb4c67ad65a7b0c1466798 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.055s user 0.002s sys 0.005s
I20240501 08:12:14.160354 31710 raft_consensus.cc:348] T c5f4c78e69fb4c67ad65a7b0c1466798 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:14.161391 31708 raft_consensus.cc:348] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:14.161674 31708 raft_consensus.cc:374] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:14.161828 31708 raft_consensus.cc:729] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:14.162940 31707 ts_tablet_manager.cc:1398] T 22cdd207ff0c49e0b89d68de068d3603 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:12:14.163424 31708 consensus_queue.cc:260] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:14.163717 31710 raft_consensus.cc:374] T c5f4c78e69fb4c67ad65a7b0c1466798 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:14.164259 31708 ts_tablet_manager.cc:1398] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:14.164420 31710 raft_consensus.cc:729] T c5f4c78e69fb4c67ad65a7b0c1466798 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:14.164808 31710 consensus_queue.cc:260] T c5f4c78e69fb4c67ad65a7b0c1466798 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:14.165288 31708 tablet_bootstrap.cc:492] T fea1af7c0ecc4ddba679a981125d957a P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:14.166129 31710 ts_tablet_manager.cc:1398] T c5f4c78e69fb4c67ad65a7b0c1466798 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240501 08:12:14.171986 31708 tablet_bootstrap.cc:654] T fea1af7c0ecc4ddba679a981125d957a P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:14.172011 31706 tablet_bootstrap.cc:492] T 22cdd207ff0c49e0b89d68de068d3603 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:14.172194 31706 ts_tablet_manager.cc:1367] T 22cdd207ff0c49e0b89d68de068d3603 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.005s
I20240501 08:12:14.173061 31706 raft_consensus.cc:348] T 22cdd207ff0c49e0b89d68de068d3603 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:14.173382 31706 raft_consensus.cc:374] T 22cdd207ff0c49e0b89d68de068d3603 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:14.173508 31706 raft_consensus.cc:729] T 22cdd207ff0c49e0b89d68de068d3603 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:14.173849 31706 consensus_queue.cc:260] T 22cdd207ff0c49e0b89d68de068d3603 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:14.174571 31706 ts_tablet_manager.cc:1398] T 22cdd207ff0c49e0b89d68de068d3603 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:14.175624 31713 tablet_bootstrap.cc:492] T 22cdd207ff0c49e0b89d68de068d3603 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:12:14.175823 31713 ts_tablet_manager.cc:1367] T 22cdd207ff0c49e0b89d68de068d3603 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.059s user 0.004s sys 0.000s
I20240501 08:12:14.176708 31713 raft_consensus.cc:348] T 22cdd207ff0c49e0b89d68de068d3603 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:14.176982 31713 raft_consensus.cc:374] T 22cdd207ff0c49e0b89d68de068d3603 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:14.177122 31713 raft_consensus.cc:729] T 22cdd207ff0c49e0b89d68de068d3603 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:12:14.177403 31713 consensus_queue.cc:260] T 22cdd207ff0c49e0b89d68de068d3603 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:14.180959 31713 ts_tablet_manager.cc:1398] T 22cdd207ff0c49e0b89d68de068d3603 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:12:14.188688 31616 raft_consensus.cc:482] T 22cdd207ff0c49e0b89d68de068d3603 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:14.188877 31616 raft_consensus.cc:504] T 22cdd207ff0c49e0b89d68de068d3603 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:14.189657 31616 leader_election.cc:288] T 22cdd207ff0c49e0b89d68de068d3603 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:14.190382 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22cdd207ff0c49e0b89d68de068d3603" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:14.190658 30672 raft_consensus.cc:2454] T 22cdd207ff0c49e0b89d68de068d3603 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:14.191187 30870 leader_election.cc:302] T 22cdd207ff0c49e0b89d68de068d3603 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a022e1a501f4d44a0468054d6c3de47, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:14.191888 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22cdd207ff0c49e0b89d68de068d3603" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:12:14.192132 27759 raft_consensus.cc:2454] T 22cdd207ff0c49e0b89d68de068d3603 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:14.192193 31712 tablet_bootstrap.cc:492] T c5f4c78e69fb4c67ad65a7b0c1466798 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:12:14.192328 31712 ts_tablet_manager.cc:1367] T c5f4c78e69fb4c67ad65a7b0c1466798 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.002s
I20240501 08:12:14.193323 31712 raft_consensus.cc:348] T c5f4c78e69fb4c67ad65a7b0c1466798 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:14.193615 31712 raft_consensus.cc:374] T c5f4c78e69fb4c67ad65a7b0c1466798 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:14.193759 31712 raft_consensus.cc:729] T c5f4c78e69fb4c67ad65a7b0c1466798 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:12:14.194145 31712 consensus_queue.cc:260] T c5f4c78e69fb4c67ad65a7b0c1466798 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:14.194870 31712 ts_tablet_manager.cc:1398] T c5f4c78e69fb4c67ad65a7b0c1466798 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:12:14.194905 31616 raft_consensus.cc:2781] T 22cdd207ff0c49e0b89d68de068d3603 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:14.195040 31616 raft_consensus.cc:482] T 22cdd207ff0c49e0b89d68de068d3603 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:14.195132 31616 raft_consensus.cc:3037] T 22cdd207ff0c49e0b89d68de068d3603 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:14.198276 31616 raft_consensus.cc:504] T 22cdd207ff0c49e0b89d68de068d3603 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:14.198560 31708 tablet_bootstrap.cc:492] T fea1af7c0ecc4ddba679a981125d957a P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:14.198568 31711 tablet_bootstrap.cc:492] T fea1af7c0ecc4ddba679a981125d957a P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:12:14.198717 31711 ts_tablet_manager.cc:1367] T fea1af7c0ecc4ddba679a981125d957a P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.048s user 0.010s sys 0.005s
I20240501 08:12:14.198765 31708 ts_tablet_manager.cc:1367] T fea1af7c0ecc4ddba679a981125d957a P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.034s user 0.023s sys 0.000s
I20240501 08:12:14.199563 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22cdd207ff0c49e0b89d68de068d3603" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:14.199812 30672 raft_consensus.cc:3037] T 22cdd207ff0c49e0b89d68de068d3603 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:14.199628 31711 raft_consensus.cc:348] T fea1af7c0ecc4ddba679a981125d957a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:14.199952 31711 raft_consensus.cc:374] T fea1af7c0ecc4ddba679a981125d957a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:14.200098 31711 raft_consensus.cc:729] T fea1af7c0ecc4ddba679a981125d957a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:12:14.200357 31708 raft_consensus.cc:348] T fea1af7c0ecc4ddba679a981125d957a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:14.200416 31711 consensus_queue.cc:260] T fea1af7c0ecc4ddba679a981125d957a P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:14.200702 31708 raft_consensus.cc:374] T fea1af7c0ecc4ddba679a981125d957a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:14.200886 31708 raft_consensus.cc:729] T fea1af7c0ecc4ddba679a981125d957a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:14.201201 31708 consensus_queue.cc:260] T fea1af7c0ecc4ddba679a981125d957a P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:14.202661 31708 ts_tablet_manager.cc:1398] T fea1af7c0ecc4ddba679a981125d957a P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:14.202673 30672 raft_consensus.cc:2454] T 22cdd207ff0c49e0b89d68de068d3603 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:14.203235 30870 leader_election.cc:302] T 22cdd207ff0c49e0b89d68de068d3603 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a022e1a501f4d44a0468054d6c3de47, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:14.203559 31615 raft_consensus.cc:2781] T 22cdd207ff0c49e0b89d68de068d3603 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:14.203686 31615 raft_consensus.cc:686] T 22cdd207ff0c49e0b89d68de068d3603 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Becoming Leader. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:12:14.203941 31615 consensus_queue.cc:237] T 22cdd207ff0c49e0b89d68de068d3603 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:14.206231 31616 leader_election.cc:288] T 22cdd207ff0c49e0b89d68de068d3603 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Requested vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:14.206766 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22cdd207ff0c49e0b89d68de068d3603" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:12:14.207020 27759 raft_consensus.cc:3037] T 22cdd207ff0c49e0b89d68de068d3603 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:14.209714 27759 raft_consensus.cc:2454] T 22cdd207ff0c49e0b89d68de068d3603 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:14.226109 31711 ts_tablet_manager.cc:1398] T fea1af7c0ecc4ddba679a981125d957a P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20240501 08:12:14.226894 27617 catalog_manager.cc:5484] T 22cdd207ff0c49e0b89d68de068d3603 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: term changed from 0 to 1, leader changed from <none> to a29cd590999b4f32849ff5689cd53c11 (127.21.84.136). New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:14.228861 27620 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:14.248103 31691 tablet.cc:1722] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:14.252395 31693 tablet.cc:1722] T cfef339b34e8403bb91e17a2fc400834 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:14.254499 31705 tablet.cc:1722] T cfef339b34e8403bb91e17a2fc400834 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:14.269707 27617 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table_new_name [id=f5fae2fca0e04b36af07b0519bdc94c0] alter complete (version 3)
I20240501 08:12:14.333763 31562 raft_consensus.cc:482] T fea1af7c0ecc4ddba679a981125d957a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:14.334021 31562 raft_consensus.cc:504] T fea1af7c0ecc4ddba679a981125d957a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:14.334689 31562 leader_election.cc:288] T fea1af7c0ecc4ddba679a981125d957a P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:12:14.335325 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fea1af7c0ecc4ddba679a981125d957a" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:14.335394 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fea1af7c0ecc4ddba679a981125d957a" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:12:14.335585 30672 raft_consensus.cc:2454] T fea1af7c0ecc4ddba679a981125d957a P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:14.335602 27890 raft_consensus.cc:2454] T fea1af7c0ecc4ddba679a981125d957a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:14.336269 30737 leader_election.cc:302] T fea1af7c0ecc4ddba679a981125d957a P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:14.336858 31562 raft_consensus.cc:2781] T fea1af7c0ecc4ddba679a981125d957a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:14.336999 31562 raft_consensus.cc:482] T fea1af7c0ecc4ddba679a981125d957a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:14.337103 31562 raft_consensus.cc:3037] T fea1af7c0ecc4ddba679a981125d957a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:14.339385 31562 raft_consensus.cc:504] T fea1af7c0ecc4ddba679a981125d957a P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:14.340947 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fea1af7c0ecc4ddba679a981125d957a" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:12:14.341056 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fea1af7c0ecc4ddba679a981125d957a" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:14.341183 27890 raft_consensus.cc:3037] T fea1af7c0ecc4ddba679a981125d957a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:14.341287 30672 raft_consensus.cc:3037] T fea1af7c0ecc4ddba679a981125d957a P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:14.344291 30672 raft_consensus.cc:2454] T fea1af7c0ecc4ddba679a981125d957a P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:14.344525 31562 leader_election.cc:288] T fea1af7c0ecc4ddba679a981125d957a P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Requested vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:12:14.344775 30737 leader_election.cc:302] T fea1af7c0ecc4ddba679a981125d957a P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:14.345724 27890 raft_consensus.cc:2454] T fea1af7c0ecc4ddba679a981125d957a P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:14.346064 31562 raft_consensus.cc:2781] T fea1af7c0ecc4ddba679a981125d957a P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:14.346520 31562 raft_consensus.cc:686] T fea1af7c0ecc4ddba679a981125d957a P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Becoming Leader. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Running, Role: LEADER
I20240501 08:12:14.347082 31562 consensus_queue.cc:237] T fea1af7c0ecc4ddba679a981125d957a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:14.350693 27611 catalog_manager.cc:5484] T fea1af7c0ecc4ddba679a981125d957a P 64998323a1f4482d8704bf20594d13a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64998323a1f4482d8704bf20594d13a5 (127.21.84.135). New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:14.375362 27611 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:14.375587 27611 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:14.378691 27611 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet cfef339b34e8403bb91e17a2fc400834
I20240501 08:12:14.379531 30917 tablet_service.cc:1514] Processing DeleteTablet for tablet cfef339b34e8403bb91e17a2fc400834 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:14 UTC) from {username='slave'} at 127.0.0.1:54036
I20240501 08:12:14.384722 30652 tablet_service.cc:1514] Processing DeleteTablet for tablet cfef339b34e8403bb91e17a2fc400834 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:14 UTC) from {username='slave'} at 127.0.0.1:57156
I20240501 08:12:14.388042 28002 tablet_service.cc:1514] Processing DeleteTablet for tablet cfef339b34e8403bb91e17a2fc400834 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:14 UTC) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:14.388690 31727 tablet_replica.cc:330] T cfef339b34e8403bb91e17a2fc400834 P 9a022e1a501f4d44a0468054d6c3de47: stopping tablet replica
I20240501 08:12:14.388990 31727 raft_consensus.cc:2229] T cfef339b34e8403bb91e17a2fc400834 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:14.389322 31727 raft_consensus.cc:2258] T cfef339b34e8403bb91e17a2fc400834 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:14.393859 31727 ts_tablet_manager.cc:1875] T cfef339b34e8403bb91e17a2fc400834 P 9a022e1a501f4d44a0468054d6c3de47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:14.401860 31727 ts_tablet_manager.cc:1888] T cfef339b34e8403bb91e17a2fc400834 P 9a022e1a501f4d44a0468054d6c3de47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:14.402215 31727 log.cc:1196] T cfef339b34e8403bb91e17a2fc400834 P 9a022e1a501f4d44a0468054d6c3de47: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/wals/cfef339b34e8403bb91e17a2fc400834
I20240501 08:12:14.402204 31728 tablet_replica.cc:330] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:14.402458 31728 raft_consensus.cc:2229] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:14.403050 31728 raft_consensus.cc:2258] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:14.404016 31728 ts_tablet_manager.cc:1875] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:14.406397 31727 ts_tablet_manager.cc:1909] T cfef339b34e8403bb91e17a2fc400834 P 9a022e1a501f4d44a0468054d6c3de47: Deleting consensus metadata
I20240501 08:12:14.406901 31726 tablet_replica.cc:330] T cfef339b34e8403bb91e17a2fc400834 P a29cd590999b4f32849ff5689cd53c11: stopping tablet replica
I20240501 08:12:14.407230 31726 raft_consensus.cc:2229] T cfef339b34e8403bb91e17a2fc400834 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:14.407564 31726 raft_consensus.cc:2258] T cfef339b34e8403bb91e17a2fc400834 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:14.407991 27604 catalog_manager.cc:4830] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): tablet cfef339b34e8403bb91e17a2fc400834 (table rebalancer_dynamic_table_new_name [id=f5fae2fca0e04b36af07b0519bdc94c0]) successfully deleted
I20240501 08:12:14.409039 31726 ts_tablet_manager.cc:1875] T cfef339b34e8403bb91e17a2fc400834 P a29cd590999b4f32849ff5689cd53c11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:14.413877 31728 ts_tablet_manager.cc:1888] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:14.414040 31728 log.cc:1196] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/cfef339b34e8403bb91e17a2fc400834
I20240501 08:12:14.414928 31728 ts_tablet_manager.cc:1909] T cfef339b34e8403bb91e17a2fc400834 P aad81047009a4ffcbc450b60030583b5: Deleting consensus metadata
I20240501 08:12:14.416275 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet cfef339b34e8403bb91e17a2fc400834 (table rebalancer_dynamic_table_new_name [id=f5fae2fca0e04b36af07b0519bdc94c0]) successfully deleted
I20240501 08:12:14.419857 31726 ts_tablet_manager.cc:1888] T cfef339b34e8403bb91e17a2fc400834 P a29cd590999b4f32849ff5689cd53c11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:14.420037 31726 log.cc:1196] T cfef339b34e8403bb91e17a2fc400834 P a29cd590999b4f32849ff5689cd53c11: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/wals/cfef339b34e8403bb91e17a2fc400834
I20240501 08:12:14.420545 31726 ts_tablet_manager.cc:1909] T cfef339b34e8403bb91e17a2fc400834 P a29cd590999b4f32849ff5689cd53c11: Deleting consensus metadata
I20240501 08:12:14.421875 27606 catalog_manager.cc:4830] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): tablet cfef339b34e8403bb91e17a2fc400834 (table rebalancer_dynamic_table_new_name [id=f5fae2fca0e04b36af07b0519bdc94c0]) successfully deleted
I20240501 08:12:14.489701 27620 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:14.497319 30917 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8cb609c91c41e4b873655cad2793ed (DEFAULT_TABLE table=rebalancer_dynamic_table [id=64d293ca12a34db38138e291bf497da0]), partition=
I20240501 08:12:14.497563 30652 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8cb609c91c41e4b873655cad2793ed (DEFAULT_TABLE table=rebalancer_dynamic_table [id=64d293ca12a34db38138e291bf497da0]), partition=
I20240501 08:12:14.497761 30917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c8cb609c91c41e4b873655cad2793ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:14.497959 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c8cb609c91c41e4b873655cad2793ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:14.498154 30514 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8cb609c91c41e4b873655cad2793ed (DEFAULT_TABLE table=rebalancer_dynamic_table [id=64d293ca12a34db38138e291bf497da0]), partition=
I20240501 08:12:14.498484 30514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c8cb609c91c41e4b873655cad2793ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:14.503226 31707 tablet_bootstrap.cc:492] T 6c8cb609c91c41e4b873655cad2793ed P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:14.504197 31706 tablet_bootstrap.cc:492] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:14.504655 31707 tablet_bootstrap.cc:654] T 6c8cb609c91c41e4b873655cad2793ed P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:14.505741 31706 tablet_bootstrap.cc:654] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:14.506562 31707 tablet_bootstrap.cc:492] T 6c8cb609c91c41e4b873655cad2793ed P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:14.506733 31707 ts_tablet_manager.cc:1367] T 6c8cb609c91c41e4b873655cad2793ed P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:12:14.507153 31710 tablet_bootstrap.cc:492] T 6c8cb609c91c41e4b873655cad2793ed P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:14.507478 31707 raft_consensus.cc:348] T 6c8cb609c91c41e4b873655cad2793ed P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:14.507701 31707 raft_consensus.cc:374] T 6c8cb609c91c41e4b873655cad2793ed P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:14.507792 31707 raft_consensus.cc:729] T 6c8cb609c91c41e4b873655cad2793ed P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:14.509294 31710 tablet_bootstrap.cc:654] T 6c8cb609c91c41e4b873655cad2793ed P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:14.509814 31706 tablet_bootstrap.cc:492] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:14.510039 31706 ts_tablet_manager.cc:1367] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240501 08:12:14.510272 31707 consensus_queue.cc:260] T 6c8cb609c91c41e4b873655cad2793ed P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:14.510788 31706 raft_consensus.cc:348] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:14.511004 31706 raft_consensus.cc:374] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:14.511075 31707 ts_tablet_manager.cc:1398] T 6c8cb609c91c41e4b873655cad2793ed P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 08:12:14.511094 31706 raft_consensus.cc:729] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:14.511307 31706 consensus_queue.cc:260] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:14.511811 31706 ts_tablet_manager.cc:1398] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:14.512785 31710 tablet_bootstrap.cc:492] T 6c8cb609c91c41e4b873655cad2793ed P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:14.513010 31710 ts_tablet_manager.cc:1367] T 6c8cb609c91c41e4b873655cad2793ed P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 08:12:14.513886 31710 raft_consensus.cc:348] T 6c8cb609c91c41e4b873655cad2793ed P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:14.514272 31710 raft_consensus.cc:374] T 6c8cb609c91c41e4b873655cad2793ed P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:14.514384 31710 raft_consensus.cc:729] T 6c8cb609c91c41e4b873655cad2793ed P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:14.514689 31710 consensus_queue.cc:260] T 6c8cb609c91c41e4b873655cad2793ed P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:14.515260 31710 ts_tablet_manager.cc:1398] T 6c8cb609c91c41e4b873655cad2793ed P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:14.554524 31562 raft_consensus.cc:482] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:14.554761 31562 raft_consensus.cc:504] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:14.555397 31562 leader_election.cc:288] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:14.555997 28019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5f4c78e69fb4c67ad65a7b0c1466798" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:12:14.556017 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5f4c78e69fb4c67ad65a7b0c1466798" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:14.556245 28019 raft_consensus.cc:2454] T c5f4c78e69fb4c67ad65a7b0c1466798 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:14.556270 30537 raft_consensus.cc:2454] T c5f4c78e69fb4c67ad65a7b0c1466798 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:14.557010 30739 leader_election.cc:302] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:12:14.557408 31562 raft_consensus.cc:2781] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:14.557507 31562 raft_consensus.cc:482] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:14.557595 31562 raft_consensus.cc:3037] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:14.560505 31562 raft_consensus.cc:504] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:14.561231 31562 leader_election.cc:288] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Requested vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:14.562747 28019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5f4c78e69fb4c67ad65a7b0c1466798" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:12:14.562991 28019 raft_consensus.cc:3037] T c5f4c78e69fb4c67ad65a7b0c1466798 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:14.562970 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5f4c78e69fb4c67ad65a7b0c1466798" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:14.563194 30537 raft_consensus.cc:3037] T c5f4c78e69fb4c67ad65a7b0c1466798 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:14.565773 30537 raft_consensus.cc:2454] T c5f4c78e69fb4c67ad65a7b0c1466798 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:14.566085 28019 raft_consensus.cc:2454] T c5f4c78e69fb4c67ad65a7b0c1466798 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:14.566269 30740 leader_election.cc:302] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 64998323a1f4482d8704bf20594d13a5; no voters:
I20240501 08:12:14.566651 31562 raft_consensus.cc:2781] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:14.566813 31562 raft_consensus.cc:686] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Becoming Leader. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Running, Role: LEADER
I20240501 08:12:14.567075 31562 consensus_queue.cc:237] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:14.570416 27620 catalog_manager.cc:5484] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64998323a1f4482d8704bf20594d13a5 (127.21.84.135). New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:14.584803 31723 raft_consensus.cc:482] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:14.585000 31723 raft_consensus.cc:504] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:14.585762 31723 leader_election.cc:288] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:14.586146 30935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8cb609c91c41e4b873655cad2793ed" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:14.586294 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8cb609c91c41e4b873655cad2793ed" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:14.586406 30935 raft_consensus.cc:2454] T 6c8cb609c91c41e4b873655cad2793ed P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 0.
I20240501 08:12:14.586552 30537 raft_consensus.cc:2454] T 6c8cb609c91c41e4b873655cad2793ed P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 0.
I20240501 08:12:14.586884 30607 leader_election.cc:302] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a022e1a501f4d44a0468054d6c3de47, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:14.587158 31723 raft_consensus.cc:2781] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:14.587276 31723 raft_consensus.cc:482] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:14.587350 31723 raft_consensus.cc:3037] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:14.590062 31723 raft_consensus.cc:504] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:14.590642 31723 leader_election.cc:288] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 election: Requested vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:14.591096 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8cb609c91c41e4b873655cad2793ed" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:14.591117 30935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8cb609c91c41e4b873655cad2793ed" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:14.591317 30935 raft_consensus.cc:3037] T 6c8cb609c91c41e4b873655cad2793ed P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:14.591328 30537 raft_consensus.cc:3037] T 6c8cb609c91c41e4b873655cad2793ed P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:14.594257 30537 raft_consensus.cc:2454] T 6c8cb609c91c41e4b873655cad2793ed P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 1.
I20240501 08:12:14.594729 30608 leader_election.cc:302] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:14.595064 31723 raft_consensus.cc:2781] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:14.595458 31723 raft_consensus.cc:686] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Becoming Leader. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Running, Role: LEADER
I20240501 08:12:14.595911 30935 raft_consensus.cc:2454] T 6c8cb609c91c41e4b873655cad2793ed P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 1.
I20240501 08:12:14.596014 31723 consensus_queue.cc:237] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:14.600317 27620 catalog_manager.cc:5484] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134). New cstate: current_term: 1 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:14.623812 31615 consensus_queue.cc:1035] T 22cdd207ff0c49e0b89d68de068d3603 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:14.628757 31615 consensus_queue.cc:1035] T 22cdd207ff0c49e0b89d68de068d3603 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:14.711123 27617 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:14.717332 30537 raft_consensus.cc:1261] T 6c8cb609c91c41e4b873655cad2793ed P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:14.717343 30935 raft_consensus.cc:1261] T 6c8cb609c91c41e4b873655cad2793ed P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:14.718041 31723 consensus_queue.cc:1035] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:14.718457 31592 consensus_queue.cc:1035] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:14.723062 27617 catalog_manager.cc:5576] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845) does not have the latest schema for tablet 6c8cb609c91c41e4b873655cad2793ed (table rebalancer_dynamic_table [id=64d293ca12a34db38138e291bf497da0]). Expected version 1 got 0
I20240501 08:12:14.723595 31705 tablet.cc:1722] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:14.724140 31705 tablet.cc:1620] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:14.725391 27617 catalog_manager.cc:5576] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933) does not have the latest schema for tablet 6c8cb609c91c41e4b873655cad2793ed (table rebalancer_dynamic_table [id=64d293ca12a34db38138e291bf497da0]). Expected version 1 got 0
I20240501 08:12:14.726675 31693 tablet.cc:1722] T 6c8cb609c91c41e4b873655cad2793ed P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:14.727234 31693 tablet.cc:1620] T 6c8cb609c91c41e4b873655cad2793ed P a29cd590999b4f32849ff5689cd53c11: MemRowSet was empty: no flush needed.
I20240501 08:12:14.733641 27617 catalog_manager.cc:5576] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293) does not have the latest schema for tablet 6c8cb609c91c41e4b873655cad2793ed (table rebalancer_dynamic_table [id=64d293ca12a34db38138e291bf497da0]). Expected version 1 got 0
I20240501 08:12:14.734189 27611 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=64d293ca12a34db38138e291bf497da0] alter complete (version 1)
I20240501 08:12:14.735034 31740 tablet.cc:1722] T 6c8cb609c91c41e4b873655cad2793ed P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:14.736737 31740 tablet.cc:1620] T 6c8cb609c91c41e4b873655cad2793ed P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:14.746852 31562 consensus_queue.cc:1035] T fea1af7c0ecc4ddba679a981125d957a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:14.752148 31562 consensus_queue.cc:1035] T fea1af7c0ecc4ddba679a981125d957a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:14.839264 27613 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:14.847371 31705 tablet.cc:1722] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:14.847950 31705 tablet.cc:1620] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:14.847950 31740 tablet.cc:1722] T 6c8cb609c91c41e4b873655cad2793ed P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:14.848376 31740 tablet.cc:1620] T 6c8cb609c91c41e4b873655cad2793ed P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:14.848441 31693 tablet.cc:1722] T 6c8cb609c91c41e4b873655cad2793ed P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:14.848934 31693 tablet.cc:1620] T 6c8cb609c91c41e4b873655cad2793ed P a29cd590999b4f32849ff5689cd53c11: MemRowSet was empty: no flush needed.
I20240501 08:12:14.858536 27613 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=64d293ca12a34db38138e291bf497da0] alter complete (version 2)
I20240501 08:12:14.963549 27611 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:14.970824 31705 tablet.cc:1722] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:14.971869 31693 tablet.cc:1722] T 6c8cb609c91c41e4b873655cad2793ed P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:14.972092 31740 tablet.cc:1722] T 6c8cb609c91c41e4b873655cad2793ed P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:14.980422 27611 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table_new_name [id=64d293ca12a34db38138e291bf497da0] alter complete (version 3)
I20240501 08:12:15.013303 31743 consensus_queue.cc:1035] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:15.017976 31748 consensus_queue.cc:1035] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:15.082840 27613 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:15.083094 27613 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:15.086081 27613 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 6c8cb609c91c41e4b873655cad2793ed
I20240501 08:12:15.086944 30917 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c8cb609c91c41e4b873655cad2793ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:15 UTC) from {username='slave'} at 127.0.0.1:54036
I20240501 08:12:15.087368 30514 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c8cb609c91c41e4b873655cad2793ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:15 UTC) from {username='slave'} at 127.0.0.1:45078
I20240501 08:12:15.087945 31752 tablet_replica.cc:330] T 6c8cb609c91c41e4b873655cad2793ed P 31811708c6774037b82df44024b229b6: stopping tablet replica
I20240501 08:12:15.087993 30652 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c8cb609c91c41e4b873655cad2793ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:15 UTC) from {username='slave'} at 127.0.0.1:57156
I20240501 08:12:15.088202 31752 raft_consensus.cc:2229] T 6c8cb609c91c41e4b873655cad2793ed P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:15.088410 31752 raft_consensus.cc:2258] T 6c8cb609c91c41e4b873655cad2793ed P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:15.089403 31752 ts_tablet_manager.cc:1875] T 6c8cb609c91c41e4b873655cad2793ed P 31811708c6774037b82df44024b229b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:15.096820 31752 ts_tablet_manager.cc:1888] T 6c8cb609c91c41e4b873655cad2793ed P 31811708c6774037b82df44024b229b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:15.097119 31752 log.cc:1196] T 6c8cb609c91c41e4b873655cad2793ed P 31811708c6774037b82df44024b229b6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/wals/6c8cb609c91c41e4b873655cad2793ed
I20240501 08:12:15.097208 31751 tablet_replica.cc:330] T 6c8cb609c91c41e4b873655cad2793ed P a29cd590999b4f32849ff5689cd53c11: stopping tablet replica
I20240501 08:12:15.097632 31751 raft_consensus.cc:2229] T 6c8cb609c91c41e4b873655cad2793ed P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:15.097766 31752 ts_tablet_manager.cc:1909] T 6c8cb609c91c41e4b873655cad2793ed P 31811708c6774037b82df44024b229b6: Deleting consensus metadata
I20240501 08:12:15.098079 31751 raft_consensus.cc:2258] T 6c8cb609c91c41e4b873655cad2793ed P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:15.098979 27607 catalog_manager.cc:4830] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): tablet 6c8cb609c91c41e4b873655cad2793ed (table rebalancer_dynamic_table_new_name [id=64d293ca12a34db38138e291bf497da0]) successfully deleted
I20240501 08:12:15.099380 31751 ts_tablet_manager.cc:1875] T 6c8cb609c91c41e4b873655cad2793ed P a29cd590999b4f32849ff5689cd53c11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:15.101619 31753 tablet_replica.cc:330] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47: stopping tablet replica
I20240501 08:12:15.101855 31753 raft_consensus.cc:2229] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:15.102239 31753 raft_consensus.cc:2258] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:15.104627 31753 ts_tablet_manager.cc:1875] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:15.106423 31751 ts_tablet_manager.cc:1888] T 6c8cb609c91c41e4b873655cad2793ed P a29cd590999b4f32849ff5689cd53c11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:15.106557 31751 log.cc:1196] T 6c8cb609c91c41e4b873655cad2793ed P a29cd590999b4f32849ff5689cd53c11: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/wals/6c8cb609c91c41e4b873655cad2793ed
I20240501 08:12:15.107026 31751 ts_tablet_manager.cc:1909] T 6c8cb609c91c41e4b873655cad2793ed P a29cd590999b4f32849ff5689cd53c11: Deleting consensus metadata
I20240501 08:12:15.108315 27606 catalog_manager.cc:4830] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): tablet 6c8cb609c91c41e4b873655cad2793ed (table rebalancer_dynamic_table_new_name [id=64d293ca12a34db38138e291bf497da0]) successfully deleted
I20240501 08:12:15.110495 31753 ts_tablet_manager.cc:1888] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:15.110602 31753 log.cc:1196] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/wals/6c8cb609c91c41e4b873655cad2793ed
I20240501 08:12:15.110991 31753 ts_tablet_manager.cc:1909] T 6c8cb609c91c41e4b873655cad2793ed P 9a022e1a501f4d44a0468054d6c3de47: Deleting consensus metadata
I20240501 08:12:15.111863 27604 catalog_manager.cc:4830] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): tablet 6c8cb609c91c41e4b873655cad2793ed (table rebalancer_dynamic_table_new_name [id=64d293ca12a34db38138e291bf497da0]) successfully deleted
I20240501 08:12:15.158651 27613 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_extra_table_22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:15.168283 30784 tablet_service.cc:1467] Processing CreateTablet for tablet f607d2e086cc4352a05a5ef2b02788cc (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=dbd065db73d1428c8fe77bd9af3302f6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:15.168664 30651 tablet_service.cc:1467] Processing CreateTablet for tablet f607d2e086cc4352a05a5ef2b02788cc (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=dbd065db73d1428c8fe77bd9af3302f6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:15.168802 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f607d2e086cc4352a05a5ef2b02788cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.169103 30651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f607d2e086cc4352a05a5ef2b02788cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.169106 30650 tablet_service.cc:1467] Processing CreateTablet for tablet a1965bd993d748f6a33cf93e8a3d70bf (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=dbd065db73d1428c8fe77bd9af3302f6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:15.169432 30652 tablet_service.cc:1467] Processing CreateTablet for tablet 0481654b2e3c45e98fe51845319c25ee (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=dbd065db73d1428c8fe77bd9af3302f6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:15.169593 30650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1965bd993d748f6a33cf93e8a3d70bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.169823 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0481654b2e3c45e98fe51845319c25ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.170359 28131 tablet_service.cc:1467] Processing CreateTablet for tablet a1965bd993d748f6a33cf93e8a3d70bf (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=dbd065db73d1428c8fe77bd9af3302f6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:15.170799 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1965bd993d748f6a33cf93e8a3d70bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.175782 28002 tablet_service.cc:1467] Processing CreateTablet for tablet 0481654b2e3c45e98fe51845319c25ee (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=dbd065db73d1428c8fe77bd9af3302f6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:15.176067 30917 tablet_service.cc:1467] Processing CreateTablet for tablet 0481654b2e3c45e98fe51845319c25ee (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=dbd065db73d1428c8fe77bd9af3302f6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:15.176276 30916 tablet_service.cc:1467] Processing CreateTablet for tablet f607d2e086cc4352a05a5ef2b02788cc (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=dbd065db73d1428c8fe77bd9af3302f6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:15.176419 28002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0481654b2e3c45e98fe51845319c25ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.176502 30917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0481654b2e3c45e98fe51845319c25ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.176820 30916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f607d2e086cc4352a05a5ef2b02788cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.189358 27613 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:15.190515 30916 tablet_service.cc:1467] Processing CreateTablet for tablet a1965bd993d748f6a33cf93e8a3d70bf (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=dbd065db73d1428c8fe77bd9af3302f6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:15.191054 30916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1965bd993d748f6a33cf93e8a3d70bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.193631 31757 tablet_bootstrap.cc:492] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:15.196449 31757 tablet_bootstrap.cc:654] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:15.198994 31755 tablet_bootstrap.cc:492] T 0481654b2e3c45e98fe51845319c25ee P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:15.200381 31758 tablet_bootstrap.cc:492] T a1965bd993d748f6a33cf93e8a3d70bf P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:15.204825 31758 tablet_bootstrap.cc:654] T a1965bd993d748f6a33cf93e8a3d70bf P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:15.202502 31755 tablet_bootstrap.cc:654] T 0481654b2e3c45e98fe51845319c25ee P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:15.212569 31755 tablet_bootstrap.cc:492] T 0481654b2e3c45e98fe51845319c25ee P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:15.214282 31755 ts_tablet_manager.cc:1367] T 0481654b2e3c45e98fe51845319c25ee P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240501 08:12:15.214723 30652 tablet_service.cc:1467] Processing CreateTablet for tablet b2b9abb9df4643a682c5cd5d8cc650c6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2b3e99832a35495391bd254d6410ac54]), partition=
I20240501 08:12:15.215606 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2b9abb9df4643a682c5cd5d8cc650c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.215651 31755 raft_consensus.cc:348] T 0481654b2e3c45e98fe51845319c25ee P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:15.218366 30916 tablet_service.cc:1467] Processing CreateTablet for tablet b2b9abb9df4643a682c5cd5d8cc650c6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2b3e99832a35495391bd254d6410ac54]), partition=
I20240501 08:12:15.219003 30916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2b9abb9df4643a682c5cd5d8cc650c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.223603 28131 tablet_service.cc:1467] Processing CreateTablet for tablet b2b9abb9df4643a682c5cd5d8cc650c6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2b3e99832a35495391bd254d6410ac54]), partition=
I20240501 08:12:15.224179 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2b9abb9df4643a682c5cd5d8cc650c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.217955 31755 raft_consensus.cc:374] T 0481654b2e3c45e98fe51845319c25ee P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:15.227721 31762 tablet_bootstrap.cc:492] T 0481654b2e3c45e98fe51845319c25ee P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:15.227932 31755 raft_consensus.cc:729] T 0481654b2e3c45e98fe51845319c25ee P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:15.228554 31755 consensus_queue.cc:260] T 0481654b2e3c45e98fe51845319c25ee P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:15.229480 31756 tablet_bootstrap.cc:492] T f607d2e086cc4352a05a5ef2b02788cc P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:15.233436 31758 tablet_bootstrap.cc:492] T a1965bd993d748f6a33cf93e8a3d70bf P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:12:15.233649 31758 ts_tablet_manager.cc:1367] T a1965bd993d748f6a33cf93e8a3d70bf P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.000s
I20240501 08:12:15.236244 31755 ts_tablet_manager.cc:1398] T 0481654b2e3c45e98fe51845319c25ee P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.021s user 0.000s sys 0.002s
I20240501 08:12:15.238961 31756 tablet_bootstrap.cc:654] T f607d2e086cc4352a05a5ef2b02788cc P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:15.242121 31755 tablet_bootstrap.cc:492] T f607d2e086cc4352a05a5ef2b02788cc P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:15.244895 31755 tablet_bootstrap.cc:654] T f607d2e086cc4352a05a5ef2b02788cc P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:15.246004 31762 tablet_bootstrap.cc:654] T 0481654b2e3c45e98fe51845319c25ee P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:15.248431 31755 tablet_bootstrap.cc:492] T f607d2e086cc4352a05a5ef2b02788cc P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:15.248616 31755 ts_tablet_manager.cc:1367] T f607d2e086cc4352a05a5ef2b02788cc P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20240501 08:12:15.249488 31755 raft_consensus.cc:348] T f607d2e086cc4352a05a5ef2b02788cc P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:15.249756 31755 raft_consensus.cc:374] T f607d2e086cc4352a05a5ef2b02788cc P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:15.249843 31755 raft_consensus.cc:729] T f607d2e086cc4352a05a5ef2b02788cc P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:15.251161 31762 tablet_bootstrap.cc:492] T 0481654b2e3c45e98fe51845319c25ee P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:12:15.251427 31762 ts_tablet_manager.cc:1367] T 0481654b2e3c45e98fe51845319c25ee P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20240501 08:12:15.253209 31762 raft_consensus.cc:348] T 0481654b2e3c45e98fe51845319c25ee P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:15.253556 31762 raft_consensus.cc:374] T 0481654b2e3c45e98fe51845319c25ee P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:15.253705 31762 raft_consensus.cc:729] T 0481654b2e3c45e98fe51845319c25ee P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:12:15.254065 31762 consensus_queue.cc:260] T 0481654b2e3c45e98fe51845319c25ee P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:15.250262 31755 consensus_queue.cc:260] T f607d2e086cc4352a05a5ef2b02788cc P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:15.250972 31758 raft_consensus.cc:348] T a1965bd993d748f6a33cf93e8a3d70bf P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:15.258342 31758 raft_consensus.cc:374] T a1965bd993d748f6a33cf93e8a3d70bf P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:15.258561 31758 raft_consensus.cc:729] T a1965bd993d748f6a33cf93e8a3d70bf P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:12:15.258980 31758 consensus_queue.cc:260] T a1965bd993d748f6a33cf93e8a3d70bf P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:15.259563 31755 ts_tablet_manager.cc:1398] T f607d2e086cc4352a05a5ef2b02788cc P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20240501 08:12:15.263355 31758 ts_tablet_manager.cc:1398] T a1965bd993d748f6a33cf93e8a3d70bf P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240501 08:12:15.267783 31762 ts_tablet_manager.cc:1398] T 0481654b2e3c45e98fe51845319c25ee P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20240501 08:12:15.276643 31755 tablet_bootstrap.cc:492] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:15.279040 31755 tablet_bootstrap.cc:654] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:15.282423 31757 tablet_bootstrap.cc:492] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:15.282673 31757 ts_tablet_manager.cc:1367] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.089s user 0.032s sys 0.044s
I20240501 08:12:15.284060 31757 raft_consensus.cc:348] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:15.284370 31757 raft_consensus.cc:374] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:15.284512 31757 raft_consensus.cc:729] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:15.284827 31757 consensus_queue.cc:260] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:15.286065 31758 tablet_bootstrap.cc:492] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:15.286383 31757 ts_tablet_manager.cc:1398] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:15.288395 31755 tablet_bootstrap.cc:492] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:15.288331 31756 tablet_bootstrap.cc:492] T f607d2e086cc4352a05a5ef2b02788cc P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:15.288451 31758 tablet_bootstrap.cc:654] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:15.288545 31755 ts_tablet_manager.cc:1367] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20240501 08:12:15.288547 31756 ts_tablet_manager.cc:1367] T f607d2e086cc4352a05a5ef2b02788cc P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.059s user 0.004s sys 0.023s
I20240501 08:12:15.289337 31755 raft_consensus.cc:348] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:15.289559 31755 raft_consensus.cc:374] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:15.289489 31756 raft_consensus.cc:348] T f607d2e086cc4352a05a5ef2b02788cc P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:15.289645 31755 raft_consensus.cc:729] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:15.289690 31756 raft_consensus.cc:374] T f607d2e086cc4352a05a5ef2b02788cc P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:15.289769 31756 raft_consensus.cc:729] T f607d2e086cc4352a05a5ef2b02788cc P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:15.290207 31756 consensus_queue.cc:260] T f607d2e086cc4352a05a5ef2b02788cc P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:15.290263 31755 consensus_queue.cc:260] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:15.292658 31755 ts_tablet_manager.cc:1398] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 08:12:15.293004 31755 tablet_bootstrap.cc:492] T b2b9abb9df4643a682c5cd5d8cc650c6 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:15.295241 31755 tablet_bootstrap.cc:654] T b2b9abb9df4643a682c5cd5d8cc650c6 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:15.298988 31756 ts_tablet_manager.cc:1398] T f607d2e086cc4352a05a5ef2b02788cc P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240501 08:12:15.300489 31616 raft_consensus.cc:482] T 0481654b2e3c45e98fe51845319c25ee P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:15.300659 31616 raft_consensus.cc:504] T 0481654b2e3c45e98fe51845319c25ee P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:15.301390 31616 leader_election.cc:288] T 0481654b2e3c45e98fe51845319c25ee P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:15.303490 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0481654b2e3c45e98fe51845319c25ee" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:15.307704 31755 tablet_bootstrap.cc:492] T b2b9abb9df4643a682c5cd5d8cc650c6 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:15.308269 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0481654b2e3c45e98fe51845319c25ee" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:12:15.308534 28022 raft_consensus.cc:2454] T 0481654b2e3c45e98fe51845319c25ee P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:15.309165 30872 leader_election.cc:302] T 0481654b2e3c45e98fe51845319c25ee P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a29cd590999b4f32849ff5689cd53c11, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:12:15.310022 31616 raft_consensus.cc:2781] T 0481654b2e3c45e98fe51845319c25ee P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:15.310143 31616 raft_consensus.cc:482] T 0481654b2e3c45e98fe51845319c25ee P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:15.310235 31616 raft_consensus.cc:3037] T 0481654b2e3c45e98fe51845319c25ee P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
W20240501 08:12:15.310431 30870 leader_election.cc:341] T 0481654b2e3c45e98fe51845319c25ee P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:15.312248 31756 tablet_bootstrap.cc:492] T 0481654b2e3c45e98fe51845319c25ee P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:15.312935 31616 raft_consensus.cc:504] T 0481654b2e3c45e98fe51845319c25ee P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:15.314368 30671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0481654b2e3c45e98fe51845319c25ee" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
W20240501 08:12:15.314968 30870 leader_election.cc:341] T 0481654b2e3c45e98fe51845319c25ee P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:15.315479 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0481654b2e3c45e98fe51845319c25ee" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:12:15.315693 28022 raft_consensus.cc:3037] T 0481654b2e3c45e98fe51845319c25ee P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:15.317651 31756 tablet_bootstrap.cc:654] T 0481654b2e3c45e98fe51845319c25ee P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:15.318142 31641 raft_consensus.cc:482] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:15.318332 31641 raft_consensus.cc:504] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:15.319043 31641 leader_election.cc:288] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:15.320370 31758 tablet_bootstrap.cc:492] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:12:15.320636 31758 ts_tablet_manager.cc:1367] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.035s user 0.024s sys 0.007s
I20240501 08:12:15.321429 31758 raft_consensus.cc:348] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:15.321748 31758 raft_consensus.cc:374] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:15.321882 31758 raft_consensus.cc:729] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:12:15.321856 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1965bd993d748f6a33cf93e8a3d70bf" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:12:15.322353 28150 raft_consensus.cc:2454] T a1965bd993d748f6a33cf93e8a3d70bf P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:15.322294 31758 consensus_queue.cc:260] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:15.322491 30671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1965bd993d748f6a33cf93e8a3d70bf" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:15.322845 31758 ts_tablet_manager.cc:1398] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240501 08:12:15.323177 30870 leader_election.cc:341] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:15.323400 30870 leader_election.cc:302] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, a29cd590999b4f32849ff5689cd53c11; no voters: 9a022e1a501f4d44a0468054d6c3de47
I20240501 08:12:15.323844 31756 tablet_bootstrap.cc:492] T 0481654b2e3c45e98fe51845319c25ee P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:15.324023 31756 ts_tablet_manager.cc:1367] T 0481654b2e3c45e98fe51845319c25ee P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20240501 08:12:15.324760 31756 raft_consensus.cc:348] T 0481654b2e3c45e98fe51845319c25ee P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:15.324986 31756 raft_consensus.cc:374] T 0481654b2e3c45e98fe51845319c25ee P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:15.325071 31756 raft_consensus.cc:729] T 0481654b2e3c45e98fe51845319c25ee P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:15.325347 31756 consensus_queue.cc:260] T 0481654b2e3c45e98fe51845319c25ee P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:15.326479 31641 raft_consensus.cc:2781] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:15.326603 31641 raft_consensus.cc:482] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:15.326696 31641 raft_consensus.cc:3037] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:15.327436 31756 ts_tablet_manager.cc:1398] T 0481654b2e3c45e98fe51845319c25ee P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:12:15.327741 31756 tablet_bootstrap.cc:492] T a1965bd993d748f6a33cf93e8a3d70bf P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:15.329540 31641 raft_consensus.cc:504] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:15.330020 31756 tablet_bootstrap.cc:654] T a1965bd993d748f6a33cf93e8a3d70bf P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:15.330832 28022 raft_consensus.cc:2454] T 0481654b2e3c45e98fe51845319c25ee P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:15.331454 30872 leader_election.cc:302] T 0481654b2e3c45e98fe51845319c25ee P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a29cd590999b4f32849ff5689cd53c11, aad81047009a4ffcbc450b60030583b5; no voters: 9a022e1a501f4d44a0468054d6c3de47
I20240501 08:12:15.332103 31616 leader_election.cc:288] T 0481654b2e3c45e98fe51845319c25ee P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Requested vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:15.332473 31616 raft_consensus.cc:2781] T 0481654b2e3c45e98fe51845319c25ee P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:15.332697 31616 raft_consensus.cc:686] T 0481654b2e3c45e98fe51845319c25ee P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Becoming Leader. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:12:15.333025 31616 consensus_queue.cc:237] T 0481654b2e3c45e98fe51845319c25ee P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:15.332578 31641 leader_election.cc:288] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Requested vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:15.334340 31755 ts_tablet_manager.cc:1367] T b2b9abb9df4643a682c5cd5d8cc650c6 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.009s
I20240501 08:12:15.334556 30671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1965bd993d748f6a33cf93e8a3d70bf" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:15.334877 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1965bd993d748f6a33cf93e8a3d70bf" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:12:15.335161 28150 raft_consensus.cc:3037] T a1965bd993d748f6a33cf93e8a3d70bf P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:15.335134 31755 raft_consensus.cc:348] T b2b9abb9df4643a682c5cd5d8cc650c6 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:15.335322 31755 raft_consensus.cc:374] T b2b9abb9df4643a682c5cd5d8cc650c6 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:15.335387 31755 raft_consensus.cc:729] T b2b9abb9df4643a682c5cd5d8cc650c6 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:15.335597 31755 consensus_queue.cc:260] T b2b9abb9df4643a682c5cd5d8cc650c6 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:15.336124 31755 ts_tablet_manager.cc:1398] T b2b9abb9df4643a682c5cd5d8cc650c6 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20240501 08:12:15.338361 30870 leader_election.cc:341] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:15.342491 28150 raft_consensus.cc:2454] T a1965bd993d748f6a33cf93e8a3d70bf P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:15.343214 30870 leader_election.cc:302] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, a29cd590999b4f32849ff5689cd53c11; no voters: 9a022e1a501f4d44a0468054d6c3de47
I20240501 08:12:15.343657 31641 raft_consensus.cc:2781] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:15.343834 31641 raft_consensus.cc:686] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Becoming Leader. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:12:15.344101 31641 consensus_queue.cc:237] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:15.345969 31756 tablet_bootstrap.cc:492] T a1965bd993d748f6a33cf93e8a3d70bf P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:15.346136 31756 ts_tablet_manager.cc:1367] T a1965bd993d748f6a33cf93e8a3d70bf P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.006s
I20240501 08:12:15.346787 31756 raft_consensus.cc:348] T a1965bd993d748f6a33cf93e8a3d70bf P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:15.346943 31756 raft_consensus.cc:374] T a1965bd993d748f6a33cf93e8a3d70bf P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:15.347014 31756 raft_consensus.cc:729] T a1965bd993d748f6a33cf93e8a3d70bf P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:15.347259 31756 consensus_queue.cc:260] T a1965bd993d748f6a33cf93e8a3d70bf P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:15.347250 27617 catalog_manager.cc:5484] T 0481654b2e3c45e98fe51845319c25ee P a29cd590999b4f32849ff5689cd53c11 reported cstate change: term changed from 0 to 1, leader changed from <none> to a29cd590999b4f32849ff5689cd53c11 (127.21.84.136). New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:15.347929 31756 ts_tablet_manager.cc:1398] T a1965bd993d748f6a33cf93e8a3d70bf P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:15.348289 31756 tablet_bootstrap.cc:492] T b2b9abb9df4643a682c5cd5d8cc650c6 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:15.350592 31756 tablet_bootstrap.cc:654] T b2b9abb9df4643a682c5cd5d8cc650c6 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:15.352821 31756 tablet_bootstrap.cc:492] T b2b9abb9df4643a682c5cd5d8cc650c6 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:15.352778 31743 raft_consensus.cc:482] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:15.352963 31743 raft_consensus.cc:504] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:15.352998 31756 ts_tablet_manager.cc:1367] T b2b9abb9df4643a682c5cd5d8cc650c6 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 08:12:15.353713 31743 leader_election.cc:288] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:15.353662 27617 catalog_manager.cc:5484] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 reported cstate change: term changed from 0 to 1, leader changed from <none> to a29cd590999b4f32849ff5689cd53c11 (127.21.84.136). New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:15.354311 31756 raft_consensus.cc:348] T b2b9abb9df4643a682c5cd5d8cc650c6 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:15.354521 31756 raft_consensus.cc:374] T b2b9abb9df4643a682c5cd5d8cc650c6 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:15.354472 30671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f607d2e086cc4352a05a5ef2b02788cc" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:15.354600 31756 raft_consensus.cc:729] T b2b9abb9df4643a682c5cd5d8cc650c6 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:15.354704 30671 raft_consensus.cc:2454] T f607d2e086cc4352a05a5ef2b02788cc P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:15.354854 31756 consensus_queue.cc:260] T b2b9abb9df4643a682c5cd5d8cc650c6 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:15.355217 30737 leader_election.cc:302] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:15.355583 31756 ts_tablet_manager.cc:1398] T b2b9abb9df4643a682c5cd5d8cc650c6 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:15.355593 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f607d2e086cc4352a05a5ef2b02788cc" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:15.355813 30936 raft_consensus.cc:2454] T f607d2e086cc4352a05a5ef2b02788cc P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:15.356272 31743 raft_consensus.cc:2781] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:15.356364 31743 raft_consensus.cc:482] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:15.356459 31743 raft_consensus.cc:3037] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:15.358757 31743 raft_consensus.cc:504] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:15.359875 30671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f607d2e086cc4352a05a5ef2b02788cc" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:15.360111 30671 raft_consensus.cc:3037] T f607d2e086cc4352a05a5ef2b02788cc P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:15.360545 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f607d2e086cc4352a05a5ef2b02788cc" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:15.360745 30936 raft_consensus.cc:3037] T f607d2e086cc4352a05a5ef2b02788cc P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:15.363171 30936 raft_consensus.cc:2454] T f607d2e086cc4352a05a5ef2b02788cc P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:15.363611 30739 leader_election.cc:302] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:15.363965 31562 raft_consensus.cc:2781] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:15.364121 31562 raft_consensus.cc:686] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Becoming Leader. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Running, Role: LEADER
I20240501 08:12:15.364428 31562 consensus_queue.cc:237] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:15.366011 30671 raft_consensus.cc:2454] T f607d2e086cc4352a05a5ef2b02788cc P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:15.367157 31743 leader_election.cc:288] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Requested vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:15.369640 27617 catalog_manager.cc:5484] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64998323a1f4482d8704bf20594d13a5 (127.21.84.135). New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:15.405213 31499 raft_consensus.cc:482] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:15.405390 31499 raft_consensus.cc:504] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:15.406141 31499 leader_election.cc:288] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:15.406510 30671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b9abb9df4643a682c5cd5d8cc650c6" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:15.406555 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b9abb9df4643a682c5cd5d8cc650c6" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:15.406735 30671 raft_consensus.cc:2454] T b2b9abb9df4643a682c5cd5d8cc650c6 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:12:15.406821 30936 raft_consensus.cc:2454] T b2b9abb9df4643a682c5cd5d8cc650c6 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:12:15.407117 28085 leader_election.cc:302] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:15.407385 31499 raft_consensus.cc:2781] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:15.407506 31499 raft_consensus.cc:482] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:15.407586 31499 raft_consensus.cc:3037] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:15.409896 31499 raft_consensus.cc:504] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:15.410580 31499 leader_election.cc:288] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:15.411497 30671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b9abb9df4643a682c5cd5d8cc650c6" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:15.411557 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b9abb9df4643a682c5cd5d8cc650c6" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:15.411671 30671 raft_consensus.cc:3037] T b2b9abb9df4643a682c5cd5d8cc650c6 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:15.411789 30936 raft_consensus.cc:3037] T b2b9abb9df4643a682c5cd5d8cc650c6 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:15.414069 30936 raft_consensus.cc:2454] T b2b9abb9df4643a682c5cd5d8cc650c6 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:12:15.414191 30671 raft_consensus.cc:2454] T b2b9abb9df4643a682c5cd5d8cc650c6 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:12:15.414449 28087 leader_election.cc:302] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:15.414770 31499 raft_consensus.cc:2781] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:15.414939 31499 raft_consensus.cc:686] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:12:15.415442 31499 consensus_queue.cc:237] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:15.418866 27617 catalog_manager.cc:5484] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:15.533479 27617 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:15.539431 30936 raft_consensus.cc:1261] T b2b9abb9df4643a682c5cd5d8cc650c6 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:15.539418 30671 raft_consensus.cc:1261] T b2b9abb9df4643a682c5cd5d8cc650c6 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:15.539994 31499 consensus_queue.cc:1035] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:15.540284 31754 consensus_queue.cc:1035] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:15.545749 27611 catalog_manager.cc:5576] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933) does not have the latest schema for tablet b2b9abb9df4643a682c5cd5d8cc650c6 (table rebalancer_dynamic_table [id=2b3e99832a35495391bd254d6410ac54]). Expected version 1 got 0
I20240501 08:12:15.545670 31781 tablet.cc:1722] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:15.546344 31781 tablet.cc:1620] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6: MemRowSet was empty: no flush needed.
I20240501 08:12:15.551900 27617 catalog_manager.cc:5576] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845) does not have the latest schema for tablet b2b9abb9df4643a682c5cd5d8cc650c6 (table rebalancer_dynamic_table [id=2b3e99832a35495391bd254d6410ac54]). Expected version 1 got 0
I20240501 08:12:15.552758 31783 tablet.cc:1722] T b2b9abb9df4643a682c5cd5d8cc650c6 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:15.553541 31783 tablet.cc:1620] T b2b9abb9df4643a682c5cd5d8cc650c6 P a29cd590999b4f32849ff5689cd53c11: MemRowSet was empty: no flush needed.
I20240501 08:12:15.554354 27611 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=2b3e99832a35495391bd254d6410ac54] alter complete (version 1)
I20240501 08:12:15.566653 31782 tablet.cc:1722] T b2b9abb9df4643a682c5cd5d8cc650c6 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:15.567234 31782 tablet.cc:1620] T b2b9abb9df4643a682c5cd5d8cc650c6 P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:15.571070 27617 catalog_manager.cc:5576] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933) does not have the latest schema for tablet b2b9abb9df4643a682c5cd5d8cc650c6 (table rebalancer_dynamic_table [id=2b3e99832a35495391bd254d6410ac54]). Expected version 1 got 0
I20240501 08:12:15.658084 27617 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:15.665817 31781 tablet.cc:1722] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:15.666394 31781 tablet.cc:1620] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6: MemRowSet was empty: no flush needed.
I20240501 08:12:15.666687 31783 tablet.cc:1722] T b2b9abb9df4643a682c5cd5d8cc650c6 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:15.667135 31783 tablet.cc:1620] T b2b9abb9df4643a682c5cd5d8cc650c6 P a29cd590999b4f32849ff5689cd53c11: MemRowSet was empty: no flush needed.
I20240501 08:12:15.667562 31782 tablet.cc:1722] T b2b9abb9df4643a682c5cd5d8cc650c6 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:15.667954 31782 tablet.cc:1620] T b2b9abb9df4643a682c5cd5d8cc650c6 P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:15.676883 27617 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=2b3e99832a35495391bd254d6410ac54] alter complete (version 2)
I20240501 08:12:15.762522 30672 raft_consensus.cc:3037] T 0481654b2e3c45e98fe51845319c25ee P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:15.766536 31641 consensus_queue.cc:1035] T 0481654b2e3c45e98fe51845319c25ee P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:15.768388 31562 consensus_queue.cc:1035] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:15.773065 31641 consensus_queue.cc:1035] T 0481654b2e3c45e98fe51845319c25ee P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:15.782696 27611 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:15.794979 31562 consensus_queue.cc:1035] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:15.795485 31781 tablet.cc:1722] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:15.796506 31783 tablet.cc:1722] T b2b9abb9df4643a682c5cd5d8cc650c6 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:15.808231 31782 tablet.cc:1722] T b2b9abb9df4643a682c5cd5d8cc650c6 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:15.810140 27613 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table_new_name [id=2b3e99832a35495391bd254d6410ac54] alter complete (version 3)
I20240501 08:12:15.824864 31746 raft_consensus.cc:482] T a1965bd993d748f6a33cf93e8a3d70bf P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:15.825068 31746 raft_consensus.cc:504] T a1965bd993d748f6a33cf93e8a3d70bf P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:15.825670 31746 leader_election.cc:288] T a1965bd993d748f6a33cf93e8a3d70bf P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:15.826185 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1965bd993d748f6a33cf93e8a3d70bf" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:15.826215 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1965bd993d748f6a33cf93e8a3d70bf" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:12:15.826447 28150 raft_consensus.cc:2379] T a1965bd993d748f6a33cf93e8a3d70bf P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a022e1a501f4d44a0468054d6c3de47 in current term 1: Already voted for candidate a29cd590999b4f32849ff5689cd53c11 in this term.
I20240501 08:12:15.826839 30605 leader_election.cc:302] T a1965bd993d748f6a33cf93e8a3d70bf P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a022e1a501f4d44a0468054d6c3de47; no voters: 91d11eae5ef6432ca86eb8ed446b6ea6, a29cd590999b4f32849ff5689cd53c11
I20240501 08:12:15.827145 31746 raft_consensus.cc:3037] T a1965bd993d748f6a33cf93e8a3d70bf P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:15.829149 31746 raft_consensus.cc:2726] T a1965bd993d748f6a33cf93e8a3d70bf P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:15.860683 31785 consensus_queue.cc:1035] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:15.865903 31785 consensus_queue.cc:1035] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:15.904929 27613 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_extra_table_24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:15.915191 30652 tablet_service.cc:1467] Processing CreateTablet for tablet 78ecdb4eacb5472eaa80273f58f7e123 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=2fd11fb0500a494a804ac62257c15bf4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:15.915345 30784 tablet_service.cc:1467] Processing CreateTablet for tablet 78ecdb4eacb5472eaa80273f58f7e123 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=2fd11fb0500a494a804ac62257c15bf4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:15.915704 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78ecdb4eacb5472eaa80273f58f7e123. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.915733 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78ecdb4eacb5472eaa80273f58f7e123. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.916064 28002 tablet_service.cc:1467] Processing CreateTablet for tablet 9eecbc3985624f3889d8e554cd585b11 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=2fd11fb0500a494a804ac62257c15bf4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:15.916493 28002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eecbc3985624f3889d8e554cd585b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.920531 27741 tablet_service.cc:1467] Processing CreateTablet for tablet 78ecdb4eacb5472eaa80273f58f7e123 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=2fd11fb0500a494a804ac62257c15bf4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:15.921023 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78ecdb4eacb5472eaa80273f58f7e123. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.923373 30514 tablet_service.cc:1467] Processing CreateTablet for tablet 9eecbc3985624f3889d8e554cd585b11 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=2fd11fb0500a494a804ac62257c15bf4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:15.923810 30514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eecbc3985624f3889d8e554cd585b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.927402 30651 tablet_service.cc:1467] Processing CreateTablet for tablet 8bbd50f618d14aad8e45c2aa6fbabc1f (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=2fd11fb0500a494a804ac62257c15bf4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:15.927850 30651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bbd50f618d14aad8e45c2aa6fbabc1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.930368 27872 tablet_service.cc:1467] Processing CreateTablet for tablet 9eecbc3985624f3889d8e554cd585b11 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=2fd11fb0500a494a804ac62257c15bf4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:15.930852 27872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eecbc3985624f3889d8e554cd585b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.931125 31795 tablet_bootstrap.cc:492] T 78ecdb4eacb5472eaa80273f58f7e123 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:15.931166 31796 tablet_bootstrap.cc:492] T 78ecdb4eacb5472eaa80273f58f7e123 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:15.933254 31795 tablet_bootstrap.cc:654] T 78ecdb4eacb5472eaa80273f58f7e123 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:15.933259 31796 tablet_bootstrap.cc:654] T 78ecdb4eacb5472eaa80273f58f7e123 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:15.935351 30916 tablet_service.cc:1467] Processing CreateTablet for tablet 8bbd50f618d14aad8e45c2aa6fbabc1f (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=2fd11fb0500a494a804ac62257c15bf4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:15.935820 30916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bbd50f618d14aad8e45c2aa6fbabc1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.940817 27739 tablet_service.cc:1467] Processing CreateTablet for tablet 8bbd50f618d14aad8e45c2aa6fbabc1f (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=2fd11fb0500a494a804ac62257c15bf4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:15.941359 27739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bbd50f618d14aad8e45c2aa6fbabc1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.941455 31799 tablet_bootstrap.cc:492] T 9eecbc3985624f3889d8e554cd585b11 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:15.942318 27613 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:15.942517 27613 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:15.947443 31799 tablet_bootstrap.cc:654] T 9eecbc3985624f3889d8e554cd585b11 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:15.956931 27613 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet b2b9abb9df4643a682c5cd5d8cc650c6
I20240501 08:12:15.957721 30651 tablet_service.cc:1514] Processing DeleteTablet for tablet b2b9abb9df4643a682c5cd5d8cc650c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:15 UTC) from {username='slave'} at 127.0.0.1:57156
I20240501 08:12:15.958527 31805 tablet_replica.cc:330] T b2b9abb9df4643a682c5cd5d8cc650c6 P 9a022e1a501f4d44a0468054d6c3de47: stopping tablet replica
I20240501 08:12:15.959096 30917 tablet_service.cc:1514] Processing DeleteTablet for tablet b2b9abb9df4643a682c5cd5d8cc650c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:15 UTC) from {username='slave'} at 127.0.0.1:54036
I20240501 08:12:15.959942 28131 tablet_service.cc:1514] Processing DeleteTablet for tablet b2b9abb9df4643a682c5cd5d8cc650c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:15 UTC) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:15.968262 31807 tablet_replica.cc:330] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:15.968621 31807 raft_consensus.cc:2229] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:15.969091 31807 raft_consensus.cc:2258] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:15.970412 31807 ts_tablet_manager.cc:1875] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:15.972254 31799 tablet_bootstrap.cc:492] T 9eecbc3985624f3889d8e554cd585b11 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:12:15.973453 31806 tablet_replica.cc:330] T b2b9abb9df4643a682c5cd5d8cc650c6 P a29cd590999b4f32849ff5689cd53c11: stopping tablet replica
I20240501 08:12:15.958858 31805 raft_consensus.cc:2229] T b2b9abb9df4643a682c5cd5d8cc650c6 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:15.974484 31806 raft_consensus.cc:2229] T b2b9abb9df4643a682c5cd5d8cc650c6 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:15.974818 31799 ts_tablet_manager.cc:1367] T 9eecbc3985624f3889d8e554cd585b11 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.006s
I20240501 08:12:15.973976 31805 raft_consensus.cc:2258] T b2b9abb9df4643a682c5cd5d8cc650c6 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:15.975450 31806 raft_consensus.cc:2258] T b2b9abb9df4643a682c5cd5d8cc650c6 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:15.976708 31805 ts_tablet_manager.cc:1875] T b2b9abb9df4643a682c5cd5d8cc650c6 P 9a022e1a501f4d44a0468054d6c3de47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:15.979583 31806 ts_tablet_manager.cc:1875] T b2b9abb9df4643a682c5cd5d8cc650c6 P a29cd590999b4f32849ff5689cd53c11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:15.982350 31800 tablet_bootstrap.cc:492] T 78ecdb4eacb5472eaa80273f58f7e123 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:15.982882 31802 tablet_bootstrap.cc:492] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:15.984315 31804 tablet_bootstrap.cc:492] T 9eecbc3985624f3889d8e554cd585b11 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:15.977901 31799 raft_consensus.cc:348] T 9eecbc3985624f3889d8e554cd585b11 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:15.985565 31799 raft_consensus.cc:374] T 9eecbc3985624f3889d8e554cd585b11 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:15.985749 31799 raft_consensus.cc:729] T 9eecbc3985624f3889d8e554cd585b11 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:12:15.986160 31799 consensus_queue.cc:260] T 9eecbc3985624f3889d8e554cd585b11 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:15.986662 31800 tablet_bootstrap.cc:654] T 78ecdb4eacb5472eaa80273f58f7e123 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:15.987315 31799 ts_tablet_manager.cc:1398] T 9eecbc3985624f3889d8e554cd585b11 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240501 08:12:15.989815 31796 tablet_bootstrap.cc:492] T 78ecdb4eacb5472eaa80273f58f7e123 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:15.990054 31796 ts_tablet_manager.cc:1367] T 78ecdb4eacb5472eaa80273f58f7e123 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.060s user 0.007s sys 0.013s
I20240501 08:12:15.990327 31807 ts_tablet_manager.cc:1888] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:15.990373 31795 tablet_bootstrap.cc:492] T 78ecdb4eacb5472eaa80273f58f7e123 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:15.990526 31807 log.cc:1196] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/b2b9abb9df4643a682c5cd5d8cc650c6
I20240501 08:12:15.990562 31795 ts_tablet_manager.cc:1367] T 78ecdb4eacb5472eaa80273f58f7e123 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.060s user 0.026s sys 0.012s
I20240501 08:12:15.990949 31796 raft_consensus.cc:348] T 78ecdb4eacb5472eaa80273f58f7e123 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:15.991143 31807 ts_tablet_manager.cc:1909] T b2b9abb9df4643a682c5cd5d8cc650c6 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting consensus metadata
I20240501 08:12:15.991243 31796 raft_consensus.cc:374] T 78ecdb4eacb5472eaa80273f58f7e123 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:15.991405 31796 raft_consensus.cc:729] T 78ecdb4eacb5472eaa80273f58f7e123 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:15.991319 31795 raft_consensus.cc:348] T 78ecdb4eacb5472eaa80273f58f7e123 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:15.991851 31796 consensus_queue.cc:260] T 78ecdb4eacb5472eaa80273f58f7e123 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:15.992131 31795 raft_consensus.cc:374] T 78ecdb4eacb5472eaa80273f58f7e123 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:15.992276 31795 raft_consensus.cc:729] T 78ecdb4eacb5472eaa80273f58f7e123 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:15.992555 31795 consensus_queue.cc:260] T 78ecdb4eacb5472eaa80273f58f7e123 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:15.994062 31795 ts_tablet_manager.cc:1398] T 78ecdb4eacb5472eaa80273f58f7e123 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:12:15.997138 31796 ts_tablet_manager.cc:1398] T 78ecdb4eacb5472eaa80273f58f7e123 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 08:12:15.998142 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet b2b9abb9df4643a682c5cd5d8cc650c6 (table rebalancer_dynamic_table_new_name [id=2b3e99832a35495391bd254d6410ac54]) successfully deleted
I20240501 08:12:16.001493 31802 tablet_bootstrap.cc:654] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:16.002046 31803 tablet_bootstrap.cc:492] T 9eecbc3985624f3889d8e554cd585b11 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:16.002830 31796 tablet_bootstrap.cc:492] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:16.003221 31804 tablet_bootstrap.cc:654] T 9eecbc3985624f3889d8e554cd585b11 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:16.004410 31796 tablet_bootstrap.cc:654] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:16.008567 31806 ts_tablet_manager.cc:1888] T b2b9abb9df4643a682c5cd5d8cc650c6 P a29cd590999b4f32849ff5689cd53c11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:16.009406 31805 ts_tablet_manager.cc:1888] T b2b9abb9df4643a682c5cd5d8cc650c6 P 9a022e1a501f4d44a0468054d6c3de47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:16.009560 31803 tablet_bootstrap.cc:654] T 9eecbc3985624f3889d8e554cd585b11 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:16.009701 31805 log.cc:1196] T b2b9abb9df4643a682c5cd5d8cc650c6 P 9a022e1a501f4d44a0468054d6c3de47: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/wals/b2b9abb9df4643a682c5cd5d8cc650c6
I20240501 08:12:16.010234 31805 ts_tablet_manager.cc:1909] T b2b9abb9df4643a682c5cd5d8cc650c6 P 9a022e1a501f4d44a0468054d6c3de47: Deleting consensus metadata
I20240501 08:12:16.011404 27604 catalog_manager.cc:4830] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): tablet b2b9abb9df4643a682c5cd5d8cc650c6 (table rebalancer_dynamic_table_new_name [id=2b3e99832a35495391bd254d6410ac54]) successfully deleted
I20240501 08:12:16.012187 31804 tablet_bootstrap.cc:492] T 9eecbc3985624f3889d8e554cd585b11 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:16.012234 31796 tablet_bootstrap.cc:492] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:16.012331 31804 ts_tablet_manager.cc:1367] T 9eecbc3985624f3889d8e554cd585b11 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20240501 08:12:16.012377 31796 ts_tablet_manager.cc:1367] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240501 08:12:16.013006 31796 raft_consensus.cc:348] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:16.013046 31804 raft_consensus.cc:348] T 9eecbc3985624f3889d8e554cd585b11 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:16.013201 31796 raft_consensus.cc:374] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.013270 31796 raft_consensus.cc:729] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.013504 31796 consensus_queue.cc:260] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:16.014161 31796 ts_tablet_manager.cc:1398] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:16.014449 31804 raft_consensus.cc:374] T 9eecbc3985624f3889d8e554cd585b11 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.014604 31804 raft_consensus.cc:729] T 9eecbc3985624f3889d8e554cd585b11 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.016178 31804 consensus_queue.cc:260] T 9eecbc3985624f3889d8e554cd585b11 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:16.016425 31806 log.cc:1196] T b2b9abb9df4643a682c5cd5d8cc650c6 P a29cd590999b4f32849ff5689cd53c11: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/wals/b2b9abb9df4643a682c5cd5d8cc650c6
I20240501 08:12:16.016952 31806 ts_tablet_manager.cc:1909] T b2b9abb9df4643a682c5cd5d8cc650c6 P a29cd590999b4f32849ff5689cd53c11: Deleting consensus metadata
I20240501 08:12:16.018141 31804 ts_tablet_manager.cc:1398] T 9eecbc3985624f3889d8e554cd585b11 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:12:16.019419 31802 tablet_bootstrap.cc:492] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:16.019660 31802 ts_tablet_manager.cc:1367] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.037s user 0.007s sys 0.000s
I20240501 08:12:16.020674 31802 raft_consensus.cc:348] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:16.021004 31802 raft_consensus.cc:374] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.021148 31802 raft_consensus.cc:729] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.021131 27606 catalog_manager.cc:4830] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): tablet b2b9abb9df4643a682c5cd5d8cc650c6 (table rebalancer_dynamic_table_new_name [id=2b3e99832a35495391bd254d6410ac54]) successfully deleted
I20240501 08:12:16.021508 31802 consensus_queue.cc:260] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:16.022537 31803 tablet_bootstrap.cc:492] T 9eecbc3985624f3889d8e554cd585b11 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:12:16.023116 31803 ts_tablet_manager.cc:1367] T 9eecbc3985624f3889d8e554cd585b11 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.005s
I20240501 08:12:16.023439 31802 ts_tablet_manager.cc:1398] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:16.024902 31803 raft_consensus.cc:348] T 9eecbc3985624f3889d8e554cd585b11 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:16.025104 31803 raft_consensus.cc:374] T 9eecbc3985624f3889d8e554cd585b11 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.025563 31803 raft_consensus.cc:729] T 9eecbc3985624f3889d8e554cd585b11 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.025851 31803 consensus_queue.cc:260] T 9eecbc3985624f3889d8e554cd585b11 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:16.026274 31800 tablet_bootstrap.cc:492] T 78ecdb4eacb5472eaa80273f58f7e123 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:12:16.026571 31800 ts_tablet_manager.cc:1367] T 78ecdb4eacb5472eaa80273f58f7e123 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.004s
I20240501 08:12:16.027945 31803 ts_tablet_manager.cc:1398] T 9eecbc3985624f3889d8e554cd585b11 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:12:16.029068 31800 raft_consensus.cc:348] T 78ecdb4eacb5472eaa80273f58f7e123 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:16.029369 31800 raft_consensus.cc:374] T 78ecdb4eacb5472eaa80273f58f7e123 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.029506 31800 raft_consensus.cc:729] T 78ecdb4eacb5472eaa80273f58f7e123 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.029822 31800 consensus_queue.cc:260] T 78ecdb4eacb5472eaa80273f58f7e123 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:16.030579 31800 ts_tablet_manager.cc:1398] T 78ecdb4eacb5472eaa80273f58f7e123 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:16.030844 31800 tablet_bootstrap.cc:492] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:16.032119 31800 tablet_bootstrap.cc:654] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:16.033792 31800 tablet_bootstrap.cc:492] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:12:16.033982 31800 ts_tablet_manager.cc:1367] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240501 08:12:16.034677 31800 raft_consensus.cc:348] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:16.034852 31800 raft_consensus.cc:374] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.034948 31800 raft_consensus.cc:729] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.035183 31800 consensus_queue.cc:260] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:16.035687 31800 ts_tablet_manager.cc:1398] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:16.075441 27617 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:16.083703 30784 tablet_service.cc:1467] Processing CreateTablet for tablet eecfbd29dd04469da995cdeaa73f4cb3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b1a98d5ce8f2430c9864eba83af6e3d8]), partition=
I20240501 08:12:16.083894 30916 tablet_service.cc:1467] Processing CreateTablet for tablet eecfbd29dd04469da995cdeaa73f4cb3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b1a98d5ce8f2430c9864eba83af6e3d8]), partition=
I20240501 08:12:16.084070 30651 tablet_service.cc:1467] Processing CreateTablet for tablet eecfbd29dd04469da995cdeaa73f4cb3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b1a98d5ce8f2430c9864eba83af6e3d8]), partition=
I20240501 08:12:16.084149 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eecfbd29dd04469da995cdeaa73f4cb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:16.084254 30916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eecfbd29dd04469da995cdeaa73f4cb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:16.084429 30651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eecfbd29dd04469da995cdeaa73f4cb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:16.090368 31795 tablet_bootstrap.cc:492] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:16.091632 31802 tablet_bootstrap.cc:492] T eecfbd29dd04469da995cdeaa73f4cb3 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:16.091857 31796 tablet_bootstrap.cc:492] T eecfbd29dd04469da995cdeaa73f4cb3 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:16.091969 31795 tablet_bootstrap.cc:654] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:16.093089 31802 tablet_bootstrap.cc:654] T eecfbd29dd04469da995cdeaa73f4cb3 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:16.093436 31796 tablet_bootstrap.cc:654] T eecfbd29dd04469da995cdeaa73f4cb3 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:16.100668 31795 tablet_bootstrap.cc:492] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:16.100867 31795 ts_tablet_manager.cc:1367] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.001s
I20240501 08:12:16.100960 31796 tablet_bootstrap.cc:492] T eecfbd29dd04469da995cdeaa73f4cb3 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:16.101122 31796 ts_tablet_manager.cc:1367] T eecfbd29dd04469da995cdeaa73f4cb3 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20240501 08:12:16.101655 31795 raft_consensus.cc:348] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:16.101881 31795 raft_consensus.cc:374] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.101989 31795 raft_consensus.cc:729] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.102226 31795 consensus_queue.cc:260] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:16.102802 31795 ts_tablet_manager.cc:1398] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:16.106125 31796 raft_consensus.cc:348] T eecfbd29dd04469da995cdeaa73f4cb3 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:16.106473 31796 raft_consensus.cc:374] T eecfbd29dd04469da995cdeaa73f4cb3 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.106618 31796 raft_consensus.cc:729] T eecfbd29dd04469da995cdeaa73f4cb3 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.106966 31796 consensus_queue.cc:260] T eecfbd29dd04469da995cdeaa73f4cb3 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:16.108652 31796 ts_tablet_manager.cc:1398] T eecfbd29dd04469da995cdeaa73f4cb3 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240501 08:12:16.111856 31802 tablet_bootstrap.cc:492] T eecfbd29dd04469da995cdeaa73f4cb3 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:16.112195 31802 ts_tablet_manager.cc:1367] T eecfbd29dd04469da995cdeaa73f4cb3 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.019s
I20240501 08:12:16.112977 31802 raft_consensus.cc:348] T eecfbd29dd04469da995cdeaa73f4cb3 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:16.113162 31802 raft_consensus.cc:374] T eecfbd29dd04469da995cdeaa73f4cb3 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.113332 31802 raft_consensus.cc:729] T eecfbd29dd04469da995cdeaa73f4cb3 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.113585 31802 consensus_queue.cc:260] T eecfbd29dd04469da995cdeaa73f4cb3 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:16.114207 31802 ts_tablet_manager.cc:1398] T eecfbd29dd04469da995cdeaa73f4cb3 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:12:16.155004 31785 raft_consensus.cc:482] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:16.155229 31785 raft_consensus.cc:504] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:16.156114 31785 leader_election.cc:288] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:16.156535 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bbd50f618d14aad8e45c2aa6fbabc1f" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:12:16.156677 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bbd50f618d14aad8e45c2aa6fbabc1f" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:16.156809 27759 raft_consensus.cc:2454] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:16.156932 30672 raft_consensus.cc:2454] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:16.157372 30872 leader_election.cc:302] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:16.157672 31785 raft_consensus.cc:2781] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:16.157790 31785 raft_consensus.cc:482] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:16.157900 31785 raft_consensus.cc:3037] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:16.160923 31785 raft_consensus.cc:504] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:16.161490 31785 leader_election.cc:288] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Requested vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:16.162036 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bbd50f618d14aad8e45c2aa6fbabc1f" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:16.162063 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bbd50f618d14aad8e45c2aa6fbabc1f" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:12:16.162261 30672 raft_consensus.cc:3037] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:16.162286 27759 raft_consensus.cc:3037] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:16.164639 27759 raft_consensus.cc:2454] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:16.164746 30672 raft_consensus.cc:2454] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:16.165084 30872 leader_election.cc:302] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:16.165419 31785 raft_consensus.cc:2781] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:16.165553 31785 raft_consensus.cc:686] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Becoming Leader. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:12:16.165833 31785 consensus_queue.cc:237] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:16.169057 27617 catalog_manager.cc:5484] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 reported cstate change: term changed from 0 to 1, leader changed from <none> to a29cd590999b4f32849ff5689cd53c11 (127.21.84.136). New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:16.208339 31746 raft_consensus.cc:482] T 78ecdb4eacb5472eaa80273f58f7e123 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:16.208576 31746 raft_consensus.cc:504] T 78ecdb4eacb5472eaa80273f58f7e123 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:16.209244 31746 leader_election.cc:288] T 78ecdb4eacb5472eaa80273f58f7e123 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:16.209817 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ecdb4eacb5472eaa80273f58f7e123" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:12:16.209959 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ecdb4eacb5472eaa80273f58f7e123" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:16.210126 27759 raft_consensus.cc:2454] T 78ecdb4eacb5472eaa80273f58f7e123 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 0.
I20240501 08:12:16.210187 30804 raft_consensus.cc:2454] T 78ecdb4eacb5472eaa80273f58f7e123 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 0.
I20240501 08:12:16.210706 30606 leader_election.cc:302] T 78ecdb4eacb5472eaa80273f58f7e123 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:16.211017 31746 raft_consensus.cc:2781] T 78ecdb4eacb5472eaa80273f58f7e123 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:16.211122 31746 raft_consensus.cc:482] T 78ecdb4eacb5472eaa80273f58f7e123 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:16.211207 31746 raft_consensus.cc:3037] T 78ecdb4eacb5472eaa80273f58f7e123 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:16.213164 31746 raft_consensus.cc:504] T 78ecdb4eacb5472eaa80273f58f7e123 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:16.213841 31746 leader_election.cc:288] T 78ecdb4eacb5472eaa80273f58f7e123 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 election: Requested vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:16.214357 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ecdb4eacb5472eaa80273f58f7e123" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:16.214565 30804 raft_consensus.cc:3037] T 78ecdb4eacb5472eaa80273f58f7e123 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:16.217504 30804 raft_consensus.cc:2454] T 78ecdb4eacb5472eaa80273f58f7e123 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 1.
I20240501 08:12:16.217875 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ecdb4eacb5472eaa80273f58f7e123" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:12:16.217918 30606 leader_election.cc:302] T 78ecdb4eacb5472eaa80273f58f7e123 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:16.218179 27759 raft_consensus.cc:3037] T 78ecdb4eacb5472eaa80273f58f7e123 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:16.218322 31746 raft_consensus.cc:2781] T 78ecdb4eacb5472eaa80273f58f7e123 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:16.218794 31746 raft_consensus.cc:686] T 78ecdb4eacb5472eaa80273f58f7e123 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Becoming Leader. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Running, Role: LEADER
I20240501 08:12:16.219095 31746 consensus_queue.cc:237] T 78ecdb4eacb5472eaa80273f58f7e123 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:16.220413 27759 raft_consensus.cc:2454] T 78ecdb4eacb5472eaa80273f58f7e123 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 1.
I20240501 08:12:16.222473 27617 catalog_manager.cc:5484] T 78ecdb4eacb5472eaa80273f58f7e123 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134). New cstate: current_term: 1 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:16.228284 31647 raft_consensus.cc:482] T 9eecbc3985624f3889d8e554cd585b11 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:16.228498 31647 raft_consensus.cc:504] T 9eecbc3985624f3889d8e554cd585b11 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:16.229436 31647 leader_election.cc:288] T 9eecbc3985624f3889d8e554cd585b11 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:16.230247 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eecbc3985624f3889d8e554cd585b11" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:16.230396 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eecbc3985624f3889d8e554cd585b11" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:12:16.230484 30537 raft_consensus.cc:2454] T 9eecbc3985624f3889d8e554cd585b11 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:12:16.230635 28022 raft_consensus.cc:2454] T 9eecbc3985624f3889d8e554cd585b11 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:12:16.230907 27828 leader_election.cc:302] T 9eecbc3985624f3889d8e554cd585b11 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:16.231262 31647 raft_consensus.cc:2781] T 9eecbc3985624f3889d8e554cd585b11 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:16.231353 31647 raft_consensus.cc:482] T 9eecbc3985624f3889d8e554cd585b11 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:16.231458 31647 raft_consensus.cc:3037] T 9eecbc3985624f3889d8e554cd585b11 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:16.233652 31647 raft_consensus.cc:504] T 9eecbc3985624f3889d8e554cd585b11 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:16.234388 31647 leader_election.cc:288] T 9eecbc3985624f3889d8e554cd585b11 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Requested vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:16.234791 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eecbc3985624f3889d8e554cd585b11" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:12:16.234892 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eecbc3985624f3889d8e554cd585b11" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:16.235020 28022 raft_consensus.cc:3037] T 9eecbc3985624f3889d8e554cd585b11 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:16.235121 30537 raft_consensus.cc:3037] T 9eecbc3985624f3889d8e554cd585b11 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:16.237676 28022 raft_consensus.cc:2454] T 9eecbc3985624f3889d8e554cd585b11 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:12:16.237867 30537 raft_consensus.cc:2454] T 9eecbc3985624f3889d8e554cd585b11 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:12:16.238140 27827 leader_election.cc:302] T 9eecbc3985624f3889d8e554cd585b11 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aad81047009a4ffcbc450b60030583b5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:16.239876 31647 raft_consensus.cc:2781] T 9eecbc3985624f3889d8e554cd585b11 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:16.240530 31647 raft_consensus.cc:686] T 9eecbc3985624f3889d8e554cd585b11 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Becoming Leader. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Running, Role: LEADER
I20240501 08:12:16.241111 31647 consensus_queue.cc:237] T 9eecbc3985624f3889d8e554cd585b11 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:16.244937 27617 catalog_manager.cc:5484] T 9eecbc3985624f3889d8e554cd585b11 P f1525cbc82a24d13b65f136695c903af reported cstate change: term changed from 0 to 1, leader changed from <none> to f1525cbc82a24d13b65f136695c903af (127.21.84.130). New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:16.311218 31748 raft_consensus.cc:482] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:16.311451 31748 raft_consensus.cc:504] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:16.312215 31748 leader_election.cc:288] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:16.312711 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eecfbd29dd04469da995cdeaa73f4cb3" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:16.312850 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eecfbd29dd04469da995cdeaa73f4cb3" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:16.312979 30936 raft_consensus.cc:2454] T eecfbd29dd04469da995cdeaa73f4cb3 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:16.313091 30672 raft_consensus.cc:2454] T eecfbd29dd04469da995cdeaa73f4cb3 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:16.313473 30739 leader_election.cc:302] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:16.313814 31748 raft_consensus.cc:2781] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:16.313923 31748 raft_consensus.cc:482] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:16.314065 31748 raft_consensus.cc:3037] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:16.317059 31748 raft_consensus.cc:504] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:16.318169 31748 leader_election.cc:288] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Requested vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:16.320906 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eecfbd29dd04469da995cdeaa73f4cb3" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:16.321168 30936 raft_consensus.cc:3037] T eecfbd29dd04469da995cdeaa73f4cb3 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:16.321300 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eecfbd29dd04469da995cdeaa73f4cb3" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:16.321540 30672 raft_consensus.cc:3037] T eecfbd29dd04469da995cdeaa73f4cb3 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:16.323999 30936 raft_consensus.cc:2454] T eecfbd29dd04469da995cdeaa73f4cb3 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:16.324640 30739 leader_election.cc:302] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:16.324707 30672 raft_consensus.cc:2454] T eecfbd29dd04469da995cdeaa73f4cb3 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:16.325284 31743 raft_consensus.cc:2781] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:16.325443 31743 raft_consensus.cc:686] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Becoming Leader. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Running, Role: LEADER
I20240501 08:12:16.325695 31743 consensus_queue.cc:237] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:16.339535 27617 catalog_manager.cc:5484] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64998323a1f4482d8704bf20594d13a5 (127.21.84.135). New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:16.493609 27617 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:16.499138 30936 raft_consensus.cc:1261] T eecfbd29dd04469da995cdeaa73f4cb3 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:16.499150 30670 raft_consensus.cc:1261] T eecfbd29dd04469da995cdeaa73f4cb3 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:16.499722 31748 consensus_queue.cc:1035] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:16.500020 31743 consensus_queue.cc:1035] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:16.505009 27611 catalog_manager.cc:5576] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933) does not have the latest schema for tablet eecfbd29dd04469da995cdeaa73f4cb3 (table rebalancer_dynamic_table [id=b1a98d5ce8f2430c9864eba83af6e3d8]). Expected version 1 got 0
I20240501 08:12:16.506129 27613 catalog_manager.cc:5576] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845) does not have the latest schema for tablet eecfbd29dd04469da995cdeaa73f4cb3 (table rebalancer_dynamic_table [id=b1a98d5ce8f2430c9864eba83af6e3d8]). Expected version 1 got 0
I20240501 08:12:16.506726 31829 tablet.cc:1722] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:16.507493 31829 tablet.cc:1620] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5: MemRowSet was empty: no flush needed.
I20240501 08:12:16.517849 31830 tablet.cc:1722] T eecfbd29dd04469da995cdeaa73f4cb3 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:16.518458 31830 tablet.cc:1620] T eecfbd29dd04469da995cdeaa73f4cb3 P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:16.519147 31831 tablet.cc:1722] T eecfbd29dd04469da995cdeaa73f4cb3 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:16.519665 31831 tablet.cc:1620] T eecfbd29dd04469da995cdeaa73f4cb3 P a29cd590999b4f32849ff5689cd53c11: MemRowSet was empty: no flush needed.
I20240501 08:12:16.520298 27611 catalog_manager.cc:5576] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933) does not have the latest schema for tablet eecfbd29dd04469da995cdeaa73f4cb3 (table rebalancer_dynamic_table [id=b1a98d5ce8f2430c9864eba83af6e3d8]). Expected version 1 got 0
I20240501 08:12:16.521255 27613 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=b1a98d5ce8f2430c9864eba83af6e3d8] alter complete (version 1)
I20240501 08:12:16.524241 27611 catalog_manager.cc:5576] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845) does not have the latest schema for tablet eecfbd29dd04469da995cdeaa73f4cb3 (table rebalancer_dynamic_table [id=b1a98d5ce8f2430c9864eba83af6e3d8]). Expected version 1 got 0
I20240501 08:12:16.542927 31616 consensus_queue.cc:1035] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:16.550555 31641 consensus_queue.cc:1035] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:16.629515 27613 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:16.636961 31829 tablet.cc:1722] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:16.637478 31829 tablet.cc:1620] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5: MemRowSet was empty: no flush needed.
I20240501 08:12:16.637468 31830 tablet.cc:1722] T eecfbd29dd04469da995cdeaa73f4cb3 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:16.637914 31830 tablet.cc:1620] T eecfbd29dd04469da995cdeaa73f4cb3 P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:16.641183 31831 tablet.cc:1722] T eecfbd29dd04469da995cdeaa73f4cb3 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:16.643469 31831 tablet.cc:1620] T eecfbd29dd04469da995cdeaa73f4cb3 P a29cd590999b4f32849ff5689cd53c11: MemRowSet was empty: no flush needed.
I20240501 08:12:16.647693 27611 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=b1a98d5ce8f2430c9864eba83af6e3d8] alter complete (version 2)
I20240501 08:12:16.680336 31746 consensus_queue.cc:1035] T 78ecdb4eacb5472eaa80273f58f7e123 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:16.687811 31837 consensus_queue.cc:1035] T 78ecdb4eacb5472eaa80273f58f7e123 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:16.729627 31820 consensus_queue.cc:1035] T 9eecbc3985624f3889d8e554cd585b11 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:16.747128 31820 consensus_queue.cc:1035] T 9eecbc3985624f3889d8e554cd585b11 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:16.752516 27611 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:16.761092 31829 tablet.cc:1722] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:16.762336 31831 tablet.cc:1722] T eecfbd29dd04469da995cdeaa73f4cb3 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:16.763497 31830 tablet.cc:1722] T eecfbd29dd04469da995cdeaa73f4cb3 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:16.772976 27611 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table_new_name [id=b1a98d5ce8f2430c9864eba83af6e3d8] alter complete (version 3)
I20240501 08:12:16.812996 27613 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:16.821589 30514 tablet_service.cc:1467] Processing CreateTablet for tablet 0e06cf862c954e8fb1ca087ec5184497 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=444d193c9c4c4c16969e6c88e6e590c9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:16.821736 30784 tablet_service.cc:1467] Processing CreateTablet for tablet 0e06cf862c954e8fb1ca087ec5184497 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=444d193c9c4c4c16969e6c88e6e590c9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:16.822093 30514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e06cf862c954e8fb1ca087ec5184497. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:16.822084 27872 tablet_service.cc:1467] Processing CreateTablet for tablet 0e06cf862c954e8fb1ca087ec5184497 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=444d193c9c4c4c16969e6c88e6e590c9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:16.822208 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e06cf862c954e8fb1ca087ec5184497. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:16.822515 27872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e06cf862c954e8fb1ca087ec5184497. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:16.823323 30651 tablet_service.cc:1467] Processing CreateTablet for tablet e12867cd76504438abe91fbe15498230 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=444d193c9c4c4c16969e6c88e6e590c9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:16.823732 30651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e12867cd76504438abe91fbe15498230. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:16.826620 30517 tablet_service.cc:1467] Processing CreateTablet for tablet e12867cd76504438abe91fbe15498230 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=444d193c9c4c4c16969e6c88e6e590c9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:16.827641 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e12867cd76504438abe91fbe15498230. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:16.830616 27871 tablet_service.cc:1467] Processing CreateTablet for tablet e12867cd76504438abe91fbe15498230 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=444d193c9c4c4c16969e6c88e6e590c9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:16.831053 27871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e12867cd76504438abe91fbe15498230. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:16.834182 30782 tablet_service.cc:1467] Processing CreateTablet for tablet c51625843b8d44a8879fa2fdbecbee5f (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=444d193c9c4c4c16969e6c88e6e590c9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:16.834612 30782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c51625843b8d44a8879fa2fdbecbee5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:16.836719 31845 tablet_bootstrap.cc:492] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:16.839828 28131 tablet_service.cc:1467] Processing CreateTablet for tablet c51625843b8d44a8879fa2fdbecbee5f (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=444d193c9c4c4c16969e6c88e6e590c9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:16.840425 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c51625843b8d44a8879fa2fdbecbee5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:16.845510 30652 tablet_service.cc:1467] Processing CreateTablet for tablet c51625843b8d44a8879fa2fdbecbee5f (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=444d193c9c4c4c16969e6c88e6e590c9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:16.846094 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c51625843b8d44a8879fa2fdbecbee5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:16.849526 31845 tablet_bootstrap.cc:654] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:16.849740 31847 tablet_bootstrap.cc:492] T 0e06cf862c954e8fb1ca087ec5184497 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:16.852085 31847 tablet_bootstrap.cc:654] T 0e06cf862c954e8fb1ca087ec5184497 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:16.853902 31846 tablet_bootstrap.cc:492] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:16.857095 31846 tablet_bootstrap.cc:654] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:16.860265 31849 tablet_bootstrap.cc:492] T c51625843b8d44a8879fa2fdbecbee5f P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:16.861541 31845 tablet_bootstrap.cc:492] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:16.861784 31845 ts_tablet_manager.cc:1367] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.005s
I20240501 08:12:16.862706 31845 raft_consensus.cc:348] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:16.863060 31845 raft_consensus.cc:374] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.863232 31845 raft_consensus.cc:729] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.863278 31849 tablet_bootstrap.cc:654] T c51625843b8d44a8879fa2fdbecbee5f P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:16.863914 31845 consensus_queue.cc:260] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:16.865974 31845 ts_tablet_manager.cc:1398] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 08:12:16.867854 31854 tablet_bootstrap.cc:492] T e12867cd76504438abe91fbe15498230 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:16.868916 31845 tablet_bootstrap.cc:492] T c51625843b8d44a8879fa2fdbecbee5f P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:16.870175 31748 raft_consensus.cc:482] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:16.870381 31748 raft_consensus.cc:504] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:16.871186 31748 leader_election.cc:288] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:12:16.873445 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e06cf862c954e8fb1ca087ec5184497" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:16.873831 31845 tablet_bootstrap.cc:654] T c51625843b8d44a8879fa2fdbecbee5f P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:16.875453 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e06cf862c954e8fb1ca087ec5184497" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
W20240501 08:12:16.876325 30740 leader_election.cc:341] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Illegal state: must be running to vote when last-logged opid is not known
W20240501 08:12:16.876605 30740 leader_election.cc:341] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:16.876706 30740 leader_election.cc:302] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5; no voters: 31811708c6774037b82df44024b229b6, f1525cbc82a24d13b65f136695c903af
I20240501 08:12:16.877348 31748 raft_consensus.cc:2726] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:16.878273 31854 tablet_bootstrap.cc:654] T e12867cd76504438abe91fbe15498230 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:16.880733 31845 tablet_bootstrap.cc:492] T c51625843b8d44a8879fa2fdbecbee5f P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:16.881062 31845 ts_tablet_manager.cc:1367] T c51625843b8d44a8879fa2fdbecbee5f P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.004s
I20240501 08:12:16.881173 27613 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:16.881520 27613 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:16.882218 31845 raft_consensus.cc:348] T c51625843b8d44a8879fa2fdbecbee5f P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:16.882620 31845 raft_consensus.cc:374] T c51625843b8d44a8879fa2fdbecbee5f P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.882738 31845 raft_consensus.cc:729] T c51625843b8d44a8879fa2fdbecbee5f P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.883082 31845 consensus_queue.cc:260] T c51625843b8d44a8879fa2fdbecbee5f P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:16.884735 31845 ts_tablet_manager.cc:1398] T c51625843b8d44a8879fa2fdbecbee5f P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:16.885807 31854 tablet_bootstrap.cc:492] T e12867cd76504438abe91fbe15498230 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:16.886027 31854 ts_tablet_manager.cc:1367] T e12867cd76504438abe91fbe15498230 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20240501 08:12:16.886844 31854 raft_consensus.cc:348] T e12867cd76504438abe91fbe15498230 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:16.887143 31854 raft_consensus.cc:374] T e12867cd76504438abe91fbe15498230 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.887284 31854 raft_consensus.cc:729] T e12867cd76504438abe91fbe15498230 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.887593 31854 consensus_queue.cc:260] T e12867cd76504438abe91fbe15498230 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:16.889288 31854 ts_tablet_manager.cc:1398] T e12867cd76504438abe91fbe15498230 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:16.889616 31854 tablet_bootstrap.cc:492] T c51625843b8d44a8879fa2fdbecbee5f P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:16.890983 27613 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet eecfbd29dd04469da995cdeaa73f4cb3
I20240501 08:12:16.891260 31854 tablet_bootstrap.cc:654] T c51625843b8d44a8879fa2fdbecbee5f P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:16.891686 31847 tablet_bootstrap.cc:492] T 0e06cf862c954e8fb1ca087ec5184497 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:16.891857 31847 ts_tablet_manager.cc:1367] T 0e06cf862c954e8fb1ca087ec5184497 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.009s
I20240501 08:12:16.892683 31847 raft_consensus.cc:348] T 0e06cf862c954e8fb1ca087ec5184497 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:16.892982 31847 raft_consensus.cc:374] T 0e06cf862c954e8fb1ca087ec5184497 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.892994 30651 tablet_service.cc:1514] Processing DeleteTablet for tablet eecfbd29dd04469da995cdeaa73f4cb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:16 UTC) from {username='slave'} at 127.0.0.1:57156
I20240501 08:12:16.893126 31847 raft_consensus.cc:729] T 0e06cf862c954e8fb1ca087ec5184497 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.893393 31847 consensus_queue.cc:260] T 0e06cf862c954e8fb1ca087ec5184497 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:16.893921 30916 tablet_service.cc:1514] Processing DeleteTablet for tablet eecfbd29dd04469da995cdeaa73f4cb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:16 UTC) from {username='slave'} at 127.0.0.1:54036
I20240501 08:12:16.894016 31847 ts_tablet_manager.cc:1398] T 0e06cf862c954e8fb1ca087ec5184497 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240501 08:12:16.895161 31847 tablet_bootstrap.cc:492] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:16.895993 31846 tablet_bootstrap.cc:492] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:12:16.896940 31854 tablet_bootstrap.cc:492] T c51625843b8d44a8879fa2fdbecbee5f P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:16.897182 31854 ts_tablet_manager.cc:1367] T c51625843b8d44a8879fa2fdbecbee5f P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240501 08:12:16.897431 30782 tablet_service.cc:1514] Processing DeleteTablet for tablet eecfbd29dd04469da995cdeaa73f4cb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:16 UTC) from {username='slave'} at 127.0.0.1:38952
I20240501 08:12:16.897955 31854 raft_consensus.cc:348] T c51625843b8d44a8879fa2fdbecbee5f P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:16.898253 31854 raft_consensus.cc:374] T c51625843b8d44a8879fa2fdbecbee5f P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.898365 31854 raft_consensus.cc:729] T c51625843b8d44a8879fa2fdbecbee5f P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.898661 31854 consensus_queue.cc:260] T c51625843b8d44a8879fa2fdbecbee5f P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:16.899431 31854 ts_tablet_manager.cc:1398] T c51625843b8d44a8879fa2fdbecbee5f P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:12:16.899732 31846 ts_tablet_manager.cc:1367] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.008s
I20240501 08:12:16.900269 31849 tablet_bootstrap.cc:492] T c51625843b8d44a8879fa2fdbecbee5f P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:12:16.900523 31849 ts_tablet_manager.cc:1367] T c51625843b8d44a8879fa2fdbecbee5f P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.017s
I20240501 08:12:16.901816 31849 raft_consensus.cc:348] T c51625843b8d44a8879fa2fdbecbee5f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:16.902033 31847 tablet_bootstrap.cc:654] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:16.902132 31849 raft_consensus.cc:374] T c51625843b8d44a8879fa2fdbecbee5f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.902283 31849 raft_consensus.cc:729] T c51625843b8d44a8879fa2fdbecbee5f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.902580 31849 consensus_queue.cc:260] T c51625843b8d44a8879fa2fdbecbee5f P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:16.903059 31846 raft_consensus.cc:348] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:16.903278 31846 raft_consensus.cc:374] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.903430 31849 ts_tablet_manager.cc:1398] T c51625843b8d44a8879fa2fdbecbee5f P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:12:16.903427 31846 raft_consensus.cc:729] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.903659 31846 consensus_queue.cc:260] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:16.904238 31846 ts_tablet_manager.cc:1398] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20240501 08:12:16.904551 31846 tablet_bootstrap.cc:492] T e12867cd76504438abe91fbe15498230 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:16.906306 31846 tablet_bootstrap.cc:654] T e12867cd76504438abe91fbe15498230 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:16.906966 31847 tablet_bootstrap.cc:492] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:16.907179 31847 ts_tablet_manager.cc:1367] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.002s
I20240501 08:12:16.908084 31847 raft_consensus.cc:348] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:16.908350 31847 raft_consensus.cc:374] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.908433 31847 raft_consensus.cc:729] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.908773 31847 consensus_queue.cc:260] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:16.909514 31847 ts_tablet_manager.cc:1398] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:16.950575 31859 tablet_replica.cc:330] T eecfbd29dd04469da995cdeaa73f4cb3 P a29cd590999b4f32849ff5689cd53c11: stopping tablet replica
I20240501 08:12:16.951875 31860 tablet_replica.cc:330] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5: stopping tablet replica
I20240501 08:12:16.952180 31859 raft_consensus.cc:2229] T eecfbd29dd04469da995cdeaa73f4cb3 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:16.952287 31860 raft_consensus.cc:2229] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:16.952842 31860 raft_consensus.cc:2258] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:16.953168 31859 raft_consensus.cc:2258] T eecfbd29dd04469da995cdeaa73f4cb3 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:16.954442 31858 tablet_replica.cc:330] T eecfbd29dd04469da995cdeaa73f4cb3 P 9a022e1a501f4d44a0468054d6c3de47: stopping tablet replica
I20240501 08:12:16.954589 31860 ts_tablet_manager.cc:1875] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:16.954720 31858 raft_consensus.cc:2229] T eecfbd29dd04469da995cdeaa73f4cb3 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:16.954977 31858 raft_consensus.cc:2258] T eecfbd29dd04469da995cdeaa73f4cb3 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:16.955792 31858 ts_tablet_manager.cc:1875] T eecfbd29dd04469da995cdeaa73f4cb3 P 9a022e1a501f4d44a0468054d6c3de47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:16.956717 31859 ts_tablet_manager.cc:1875] T eecfbd29dd04469da995cdeaa73f4cb3 P a29cd590999b4f32849ff5689cd53c11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:16.959630 31860 ts_tablet_manager.cc:1888] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:16.959820 31860 log.cc:1196] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal/wals/eecfbd29dd04469da995cdeaa73f4cb3
I20240501 08:12:16.960208 31860 ts_tablet_manager.cc:1909] T eecfbd29dd04469da995cdeaa73f4cb3 P 64998323a1f4482d8704bf20594d13a5: Deleting consensus metadata
I20240501 08:12:16.961354 27605 catalog_manager.cc:4830] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): tablet eecfbd29dd04469da995cdeaa73f4cb3 (table rebalancer_dynamic_table_new_name [id=b1a98d5ce8f2430c9864eba83af6e3d8]) successfully deleted
I20240501 08:12:16.962455 31820 raft_consensus.cc:482] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:16.962654 31820 raft_consensus.cc:504] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:16.963418 31820 leader_election.cc:288] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:16.964076 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e06cf862c954e8fb1ca087ec5184497" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:16.964293 30804 raft_consensus.cc:2454] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:12:16.964699 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e06cf862c954e8fb1ca087ec5184497" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:16.964963 30537 raft_consensus.cc:2454] T 0e06cf862c954e8fb1ca087ec5184497 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:12:16.965343 27826 leader_election.cc:302] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:16.965659 31846 tablet_bootstrap.cc:492] T e12867cd76504438abe91fbe15498230 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:12:16.965824 31846 ts_tablet_manager.cc:1367] T e12867cd76504438abe91fbe15498230 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.061s user 0.017s sys 0.029s
I20240501 08:12:16.965920 31858 ts_tablet_manager.cc:1888] T eecfbd29dd04469da995cdeaa73f4cb3 P 9a022e1a501f4d44a0468054d6c3de47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:16.966094 31858 log.cc:1196] T eecfbd29dd04469da995cdeaa73f4cb3 P 9a022e1a501f4d44a0468054d6c3de47: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/wals/eecfbd29dd04469da995cdeaa73f4cb3
I20240501 08:12:16.966564 31858 ts_tablet_manager.cc:1909] T eecfbd29dd04469da995cdeaa73f4cb3 P 9a022e1a501f4d44a0468054d6c3de47: Deleting consensus metadata
I20240501 08:12:16.966682 31820 raft_consensus.cc:2781] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:16.966822 31820 raft_consensus.cc:482] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:16.966930 31820 raft_consensus.cc:3037] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:16.967530 31846 raft_consensus.cc:348] T e12867cd76504438abe91fbe15498230 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:16.967798 31846 raft_consensus.cc:374] T e12867cd76504438abe91fbe15498230 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.967905 31846 raft_consensus.cc:729] T e12867cd76504438abe91fbe15498230 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.968204 31846 consensus_queue.cc:260] T e12867cd76504438abe91fbe15498230 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:16.968783 31846 ts_tablet_manager.cc:1398] T e12867cd76504438abe91fbe15498230 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:16.969051 31859 ts_tablet_manager.cc:1888] T eecfbd29dd04469da995cdeaa73f4cb3 P a29cd590999b4f32849ff5689cd53c11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:16.969507 27604 catalog_manager.cc:4830] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): tablet eecfbd29dd04469da995cdeaa73f4cb3 (table rebalancer_dynamic_table_new_name [id=b1a98d5ce8f2430c9864eba83af6e3d8]) successfully deleted
I20240501 08:12:16.972095 31820 raft_consensus.cc:504] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:16.972266 31859 log.cc:1196] T eecfbd29dd04469da995cdeaa73f4cb3 P a29cd590999b4f32849ff5689cd53c11: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/wals/eecfbd29dd04469da995cdeaa73f4cb3
I20240501 08:12:16.972728 31859 ts_tablet_manager.cc:1909] T eecfbd29dd04469da995cdeaa73f4cb3 P a29cd590999b4f32849ff5689cd53c11: Deleting consensus metadata
I20240501 08:12:16.973280 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e06cf862c954e8fb1ca087ec5184497" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:16.973529 30804 raft_consensus.cc:3037] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:16.973591 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e06cf862c954e8fb1ca087ec5184497" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:16.973851 30537 raft_consensus.cc:3037] T 0e06cf862c954e8fb1ca087ec5184497 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:16.973826 27606 catalog_manager.cc:4830] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): tablet eecfbd29dd04469da995cdeaa73f4cb3 (table rebalancer_dynamic_table_new_name [id=b1a98d5ce8f2430c9864eba83af6e3d8]) successfully deleted
I20240501 08:12:16.976154 30537 raft_consensus.cc:2454] T 0e06cf862c954e8fb1ca087ec5184497 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:12:16.976248 30804 raft_consensus.cc:2454] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:12:16.976622 27828 leader_election.cc:302] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:16.976977 31647 raft_consensus.cc:2781] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:16.977123 31647 raft_consensus.cc:686] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Becoming Leader. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Running, Role: LEADER
I20240501 08:12:16.977370 31647 consensus_queue.cc:237] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:16.976752 31820 leader_election.cc:288] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Requested vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:16.981571 27611 catalog_manager.cc:5484] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af reported cstate change: term changed from 0 to 1, leader changed from <none> to f1525cbc82a24d13b65f136695c903af (127.21.84.130). New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:16.995587 27611 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:17.002935 30782 tablet_service.cc:1467] Processing CreateTablet for tablet 48f3d69c805744d7a9cdd719856df0b5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e89a38c4f5a144eeb6505874a66f6778]), partition=
I20240501 08:12:17.003276 30651 tablet_service.cc:1467] Processing CreateTablet for tablet 48f3d69c805744d7a9cdd719856df0b5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e89a38c4f5a144eeb6505874a66f6778]), partition=
I20240501 08:12:17.003382 30782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f3d69c805744d7a9cdd719856df0b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:17.003353 30514 tablet_service.cc:1467] Processing CreateTablet for tablet 48f3d69c805744d7a9cdd719856df0b5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e89a38c4f5a144eeb6505874a66f6778]), partition=
I20240501 08:12:17.003618 30651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f3d69c805744d7a9cdd719856df0b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:17.003688 30514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f3d69c805744d7a9cdd719856df0b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:17.008260 31854 tablet_bootstrap.cc:492] T 48f3d69c805744d7a9cdd719856df0b5 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:17.010015 31854 tablet_bootstrap.cc:654] T 48f3d69c805744d7a9cdd719856df0b5 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:17.011067 31847 tablet_bootstrap.cc:492] T 48f3d69c805744d7a9cdd719856df0b5 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:17.012589 31847 tablet_bootstrap.cc:654] T 48f3d69c805744d7a9cdd719856df0b5 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:17.013463 31845 tablet_bootstrap.cc:492] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:17.014725 31845 tablet_bootstrap.cc:654] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:17.015571 31847 tablet_bootstrap.cc:492] T 48f3d69c805744d7a9cdd719856df0b5 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:17.016116 31847 ts_tablet_manager.cc:1367] T 48f3d69c805744d7a9cdd719856df0b5 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240501 08:12:17.016746 31845 tablet_bootstrap.cc:492] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:17.016873 31845 ts_tablet_manager.cc:1367] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:12:17.017679 31845 raft_consensus.cc:348] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:17.017858 31845 raft_consensus.cc:374] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:17.018162 31845 raft_consensus.cc:729] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:17.018450 31845 consensus_queue.cc:260] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:17.019639 31845 ts_tablet_manager.cc:1398] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:12:17.020490 31847 raft_consensus.cc:348] T 48f3d69c805744d7a9cdd719856df0b5 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:17.020794 31847 raft_consensus.cc:374] T 48f3d69c805744d7a9cdd719856df0b5 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:17.020898 31847 raft_consensus.cc:729] T 48f3d69c805744d7a9cdd719856df0b5 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:17.021919 31847 consensus_queue.cc:260] T 48f3d69c805744d7a9cdd719856df0b5 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:17.023003 31847 ts_tablet_manager.cc:1398] T 48f3d69c805744d7a9cdd719856df0b5 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240501 08:12:17.026266 31854 tablet_bootstrap.cc:492] T 48f3d69c805744d7a9cdd719856df0b5 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:17.026445 31854 ts_tablet_manager.cc:1367] T 48f3d69c805744d7a9cdd719856df0b5 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.009s
I20240501 08:12:17.027186 31854 raft_consensus.cc:348] T 48f3d69c805744d7a9cdd719856df0b5 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:17.027352 31854 raft_consensus.cc:374] T 48f3d69c805744d7a9cdd719856df0b5 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:17.027423 31854 raft_consensus.cc:729] T 48f3d69c805744d7a9cdd719856df0b5 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:17.027655 31854 consensus_queue.cc:260] T 48f3d69c805744d7a9cdd719856df0b5 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:17.028196 31854 ts_tablet_manager.cc:1398] T 48f3d69c805744d7a9cdd719856df0b5 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:17.042063 31348 raft_consensus.cc:482] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:17.042266 31348 raft_consensus.cc:504] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:17.042929 31348 leader_election.cc:288] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:12:17.043422 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12867cd76504438abe91fbe15498230" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:17.043673 30672 raft_consensus.cc:2454] T e12867cd76504438abe91fbe15498230 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 0.
I20240501 08:12:17.044080 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12867cd76504438abe91fbe15498230" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:12:17.044284 27890 raft_consensus.cc:2454] T e12867cd76504438abe91fbe15498230 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 0.
I20240501 08:12:17.044636 30470 leader_election.cc:302] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:17.045542 31348 raft_consensus.cc:2781] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:17.045639 31348 raft_consensus.cc:482] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:17.045722 31348 raft_consensus.cc:3037] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:17.048626 31348 raft_consensus.cc:504] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:17.049651 31348 leader_election.cc:288] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 election: Requested vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:12:17.050181 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12867cd76504438abe91fbe15498230" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:17.050307 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12867cd76504438abe91fbe15498230" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:12:17.050410 30672 raft_consensus.cc:3037] T e12867cd76504438abe91fbe15498230 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:17.050484 27890 raft_consensus.cc:3037] T e12867cd76504438abe91fbe15498230 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:17.052312 27890 raft_consensus.cc:2454] T e12867cd76504438abe91fbe15498230 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 1.
I20240501 08:12:17.052732 30473 leader_election.cc:302] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:17.052995 31348 raft_consensus.cc:2781] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:17.053169 30672 raft_consensus.cc:2454] T e12867cd76504438abe91fbe15498230 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 1.
I20240501 08:12:17.053419 31348 raft_consensus.cc:686] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Becoming Leader. State: Replica: 31811708c6774037b82df44024b229b6, State: Running, Role: LEADER
I20240501 08:12:17.053704 31348 consensus_queue.cc:237] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:17.057137 27613 catalog_manager.cc:5484] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31811708c6774037b82df44024b229b6 (127.21.84.133). New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:17.113904 31748 raft_consensus.cc:482] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:17.114125 31748 raft_consensus.cc:504] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:17.114823 31748 leader_election.cc:288] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:17.116355 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f3d69c805744d7a9cdd719856df0b5" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:17.116597 30537 raft_consensus.cc:2454] T 48f3d69c805744d7a9cdd719856df0b5 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:17.117130 30740 leader_election.cc:302] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 64998323a1f4482d8704bf20594d13a5; no voters:
I20240501 08:12:17.117435 31748 raft_consensus.cc:2781] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:17.117547 31748 raft_consensus.cc:482] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:17.117617 31748 raft_consensus.cc:3037] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:17.119822 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f3d69c805744d7a9cdd719856df0b5" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:17.120402 31748 raft_consensus.cc:504] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:17.122310 31748 leader_election.cc:288] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Requested vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:17.122406 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f3d69c805744d7a9cdd719856df0b5" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:17.122489 30670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f3d69c805744d7a9cdd719856df0b5" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:17.122611 30537 raft_consensus.cc:3037] T 48f3d69c805744d7a9cdd719856df0b5 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:17.120929 30672 raft_consensus.cc:2454] T 48f3d69c805744d7a9cdd719856df0b5 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:17.125783 30537 raft_consensus.cc:2454] T 48f3d69c805744d7a9cdd719856df0b5 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:17.126186 30740 leader_election.cc:302] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31811708c6774037b82df44024b229b6, 64998323a1f4482d8704bf20594d13a5; no voters: 9a022e1a501f4d44a0468054d6c3de47
I20240501 08:12:17.126507 31748 raft_consensus.cc:2781] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:17.131124 31748 raft_consensus.cc:686] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Becoming Leader. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Running, Role: LEADER
I20240501 08:12:17.131435 31748 consensus_queue.cc:237] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:17.134902 27613 catalog_manager.cc:5484] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64998323a1f4482d8704bf20594d13a5 (127.21.84.135). New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:17.170147 31748 raft_consensus.cc:482] T c51625843b8d44a8879fa2fdbecbee5f P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:17.170364 31748 raft_consensus.cc:504] T c51625843b8d44a8879fa2fdbecbee5f P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:17.170998 31748 leader_election.cc:288] T c51625843b8d44a8879fa2fdbecbee5f P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:17.171502 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c51625843b8d44a8879fa2fdbecbee5f" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:12:17.171576 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c51625843b8d44a8879fa2fdbecbee5f" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:17.171846 30672 raft_consensus.cc:2454] T c51625843b8d44a8879fa2fdbecbee5f P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:17.171838 28151 raft_consensus.cc:2454] T c51625843b8d44a8879fa2fdbecbee5f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:17.172232 30737 leader_election.cc:302] T c51625843b8d44a8879fa2fdbecbee5f P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:17.172487 31748 raft_consensus.cc:2781] T c51625843b8d44a8879fa2fdbecbee5f P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:17.172580 31748 raft_consensus.cc:482] T c51625843b8d44a8879fa2fdbecbee5f P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:17.172689 31748 raft_consensus.cc:3037] T c51625843b8d44a8879fa2fdbecbee5f P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:17.175385 31748 raft_consensus.cc:504] T c51625843b8d44a8879fa2fdbecbee5f P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:17.175948 31748 leader_election.cc:288] T c51625843b8d44a8879fa2fdbecbee5f P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Requested vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:17.176436 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c51625843b8d44a8879fa2fdbecbee5f" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:12:17.176493 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c51625843b8d44a8879fa2fdbecbee5f" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:17.176658 28151 raft_consensus.cc:3037] T c51625843b8d44a8879fa2fdbecbee5f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:17.176740 30672 raft_consensus.cc:3037] T c51625843b8d44a8879fa2fdbecbee5f P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:17.179066 28151 raft_consensus.cc:2454] T c51625843b8d44a8879fa2fdbecbee5f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:17.179299 30672 raft_consensus.cc:2454] T c51625843b8d44a8879fa2fdbecbee5f P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:17.179491 30737 leader_election.cc:302] T c51625843b8d44a8879fa2fdbecbee5f P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:12:17.179770 31748 raft_consensus.cc:2781] T c51625843b8d44a8879fa2fdbecbee5f P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:17.179922 31748 raft_consensus.cc:686] T c51625843b8d44a8879fa2fdbecbee5f P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Becoming Leader. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Running, Role: LEADER
I20240501 08:12:17.180204 31748 consensus_queue.cc:237] T c51625843b8d44a8879fa2fdbecbee5f P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:17.183355 27613 catalog_manager.cc:5484] T c51625843b8d44a8879fa2fdbecbee5f P 64998323a1f4482d8704bf20594d13a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64998323a1f4482d8704bf20594d13a5 (127.21.84.135). New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:17.240195 27617 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:17.246023 30672 raft_consensus.cc:3037] T 48f3d69c805744d7a9cdd719856df0b5 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:17.246543 30537 raft_consensus.cc:1261] T 48f3d69c805744d7a9cdd719856df0b5 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:17.247218 31872 consensus_queue.cc:1035] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:17.253783 30672 raft_consensus.cc:1261] T 48f3d69c805744d7a9cdd719856df0b5 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:17.255604 31872 consensus_queue.cc:1035] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:17.262399 27617 catalog_manager.cc:5576] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293) does not have the latest schema for tablet 48f3d69c805744d7a9cdd719856df0b5 (table rebalancer_dynamic_table [id=e89a38c4f5a144eeb6505874a66f6778]). Expected version 1 got 0
I20240501 08:12:17.264386 27613 catalog_manager.cc:5576] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933) does not have the latest schema for tablet 48f3d69c805744d7a9cdd719856df0b5 (table rebalancer_dynamic_table [id=e89a38c4f5a144eeb6505874a66f6778]). Expected version 1 got 0
I20240501 08:12:17.267671 31829 tablet.cc:1722] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:17.268209 31829 tablet.cc:1620] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5: MemRowSet was empty: no flush needed.
I20240501 08:12:17.275364 31882 tablet.cc:1722] T 48f3d69c805744d7a9cdd719856df0b5 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:17.276583 31882 tablet.cc:1620] T 48f3d69c805744d7a9cdd719856df0b5 P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:17.277190 27611 catalog_manager.cc:5576] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293) does not have the latest schema for tablet 48f3d69c805744d7a9cdd719856df0b5 (table rebalancer_dynamic_table [id=e89a38c4f5a144eeb6505874a66f6778]). Expected version 1 got 0
I20240501 08:12:17.280680 31883 tablet.cc:1722] T 48f3d69c805744d7a9cdd719856df0b5 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:17.281236 31883 tablet.cc:1620] T 48f3d69c805744d7a9cdd719856df0b5 P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:17.281296 27617 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=e89a38c4f5a144eeb6505874a66f6778] alter complete (version 1)
I20240501 08:12:17.286932 27617 catalog_manager.cc:5576] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933) does not have the latest schema for tablet 48f3d69c805744d7a9cdd719856df0b5 (table rebalancer_dynamic_table [id=e89a38c4f5a144eeb6505874a66f6778]). Expected version 1 got 0
I20240501 08:12:17.388005 27617 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:17.395694 31829 tablet.cc:1722] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:17.396240 31829 tablet.cc:1620] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5: MemRowSet was empty: no flush needed.
I20240501 08:12:17.398290 31883 tablet.cc:1722] T 48f3d69c805744d7a9cdd719856df0b5 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:17.398793 31883 tablet.cc:1620] T 48f3d69c805744d7a9cdd719856df0b5 P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:17.399734 31882 tablet.cc:1722] T 48f3d69c805744d7a9cdd719856df0b5 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:17.400383 31882 tablet.cc:1620] T 48f3d69c805744d7a9cdd719856df0b5 P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:17.406141 27617 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=e89a38c4f5a144eeb6505874a66f6778] alter complete (version 2)
I20240501 08:12:17.437886 31820 consensus_queue.cc:1035] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:17.439461 31348 consensus_queue.cc:1035] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:17.446430 31647 consensus_queue.cc:1035] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:17.453681 31867 consensus_queue.cc:1035] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:17.509431 27617 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:17.516685 31829 tablet.cc:1722] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:17.517810 31882 tablet.cc:1722] T 48f3d69c805744d7a9cdd719856df0b5 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:17.517818 31883 tablet.cc:1722] T 48f3d69c805744d7a9cdd719856df0b5 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:17.525897 27617 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table_new_name [id=e89a38c4f5a144eeb6505874a66f6778] alter complete (version 3)
I20240501 08:12:17.629243 27613 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:17.629526 27613 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:17.632587 27613 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 48f3d69c805744d7a9cdd719856df0b5
I20240501 08:12:17.633553 30782 tablet_service.cc:1514] Processing DeleteTablet for tablet 48f3d69c805744d7a9cdd719856df0b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:17 UTC) from {username='slave'} at 127.0.0.1:38952
I20240501 08:12:17.633565 30651 tablet_service.cc:1514] Processing DeleteTablet for tablet 48f3d69c805744d7a9cdd719856df0b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:17 UTC) from {username='slave'} at 127.0.0.1:57156
I20240501 08:12:17.633962 30514 tablet_service.cc:1514] Processing DeleteTablet for tablet 48f3d69c805744d7a9cdd719856df0b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:17 UTC) from {username='slave'} at 127.0.0.1:45078
I20240501 08:12:17.634208 31890 tablet_replica.cc:330] T 48f3d69c805744d7a9cdd719856df0b5 P 9a022e1a501f4d44a0468054d6c3de47: stopping tablet replica
I20240501 08:12:17.634449 31890 raft_consensus.cc:2229] T 48f3d69c805744d7a9cdd719856df0b5 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:17.634682 31891 tablet_replica.cc:330] T 48f3d69c805744d7a9cdd719856df0b5 P 31811708c6774037b82df44024b229b6: stopping tablet replica
I20240501 08:12:17.634699 31890 raft_consensus.cc:2258] T 48f3d69c805744d7a9cdd719856df0b5 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:17.635020 31891 raft_consensus.cc:2229] T 48f3d69c805744d7a9cdd719856df0b5 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:17.635337 31891 raft_consensus.cc:2258] T 48f3d69c805744d7a9cdd719856df0b5 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:17.635607 31890 ts_tablet_manager.cc:1875] T 48f3d69c805744d7a9cdd719856df0b5 P 9a022e1a501f4d44a0468054d6c3de47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:17.636499 31891 ts_tablet_manager.cc:1875] T 48f3d69c805744d7a9cdd719856df0b5 P 31811708c6774037b82df44024b229b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:17.638350 31889 tablet_replica.cc:330] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5: stopping tablet replica
I20240501 08:12:17.638614 31889 raft_consensus.cc:2229] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:17.639034 31889 raft_consensus.cc:2258] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:17.640604 31889 ts_tablet_manager.cc:1875] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:17.641517 31890 ts_tablet_manager.cc:1888] T 48f3d69c805744d7a9cdd719856df0b5 P 9a022e1a501f4d44a0468054d6c3de47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:17.641644 31890 log.cc:1196] T 48f3d69c805744d7a9cdd719856df0b5 P 9a022e1a501f4d44a0468054d6c3de47: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/wals/48f3d69c805744d7a9cdd719856df0b5
I20240501 08:12:17.642129 31890 ts_tablet_manager.cc:1909] T 48f3d69c805744d7a9cdd719856df0b5 P 9a022e1a501f4d44a0468054d6c3de47: Deleting consensus metadata
I20240501 08:12:17.642912 31891 ts_tablet_manager.cc:1888] T 48f3d69c805744d7a9cdd719856df0b5 P 31811708c6774037b82df44024b229b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:17.643069 31891 log.cc:1196] T 48f3d69c805744d7a9cdd719856df0b5 P 31811708c6774037b82df44024b229b6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/wals/48f3d69c805744d7a9cdd719856df0b5
I20240501 08:12:17.643250 27604 catalog_manager.cc:4830] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): tablet 48f3d69c805744d7a9cdd719856df0b5 (table rebalancer_dynamic_table_new_name [id=e89a38c4f5a144eeb6505874a66f6778]) successfully deleted
I20240501 08:12:17.643568 31891 ts_tablet_manager.cc:1909] T 48f3d69c805744d7a9cdd719856df0b5 P 31811708c6774037b82df44024b229b6: Deleting consensus metadata
I20240501 08:12:17.644696 27607 catalog_manager.cc:4830] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): tablet 48f3d69c805744d7a9cdd719856df0b5 (table rebalancer_dynamic_table_new_name [id=e89a38c4f5a144eeb6505874a66f6778]) successfully deleted
I20240501 08:12:17.646590 31889 ts_tablet_manager.cc:1888] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:17.646737 31889 log.cc:1196] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal/wals/48f3d69c805744d7a9cdd719856df0b5
I20240501 08:12:17.647261 31889 ts_tablet_manager.cc:1909] T 48f3d69c805744d7a9cdd719856df0b5 P 64998323a1f4482d8704bf20594d13a5: Deleting consensus metadata
I20240501 08:12:17.648366 27605 catalog_manager.cc:4830] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): tablet 48f3d69c805744d7a9cdd719856df0b5 (table rebalancer_dynamic_table_new_name [id=e89a38c4f5a144eeb6505874a66f6778]) successfully deleted
I20240501 08:12:17.648836 31872 consensus_queue.cc:1035] T c51625843b8d44a8879fa2fdbecbee5f P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:17.655251 31872 consensus_queue.cc:1035] T c51625843b8d44a8879fa2fdbecbee5f P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:17.735972 27617 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:17.742110 27617 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:17.743736 30514 tablet_service.cc:1467] Processing CreateTablet for tablet 868644032be244eb89977028be916a57 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3a634adcce0b4a47a32f2e24acbf09ed]), partition=
I20240501 08:12:17.744179 30514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 868644032be244eb89977028be916a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:17.744240 30916 tablet_service.cc:1467] Processing CreateTablet for tablet 868644032be244eb89977028be916a57 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3a634adcce0b4a47a32f2e24acbf09ed]), partition=
I20240501 08:12:17.744546 30916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 868644032be244eb89977028be916a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:17.749864 27741 tablet_service.cc:1467] Processing CreateTablet for tablet 868644032be244eb89977028be916a57 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3a634adcce0b4a47a32f2e24acbf09ed]), partition=
I20240501 08:12:17.750314 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 868644032be244eb89977028be916a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:17.754652 30651 tablet_service.cc:1467] Processing CreateTablet for tablet fc9e0f35ebcd48b5a27c57b2485e599e (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=474d31f683724897b682ce8741ddaa19]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:17.755146 30651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc9e0f35ebcd48b5a27c57b2485e599e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:17.755511 30782 tablet_service.cc:1467] Processing CreateTablet for tablet fc9e0f35ebcd48b5a27c57b2485e599e (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=474d31f683724897b682ce8741ddaa19]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:17.755913 30782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc9e0f35ebcd48b5a27c57b2485e599e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:17.757529 28002 tablet_service.cc:1467] Processing CreateTablet for tablet 79bd880115e1444594881c95c55016d2 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=474d31f683724897b682ce8741ddaa19]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:17.758033 28002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79bd880115e1444594881c95c55016d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:17.758366 27739 tablet_service.cc:1467] Processing CreateTablet for tablet fc9e0f35ebcd48b5a27c57b2485e599e (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=474d31f683724897b682ce8741ddaa19]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:17.758824 27739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc9e0f35ebcd48b5a27c57b2485e599e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:17.760960 30917 tablet_service.cc:1467] Processing CreateTablet for tablet 2518578063474b668508d644a2098331 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=474d31f683724897b682ce8741ddaa19]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:17.761402 30917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2518578063474b668508d644a2098331. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:17.762248 30517 tablet_service.cc:1467] Processing CreateTablet for tablet 2518578063474b668508d644a2098331 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=474d31f683724897b682ce8741ddaa19]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:17.762660 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2518578063474b668508d644a2098331. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:17.767669 30784 tablet_service.cc:1467] Processing CreateTablet for tablet 79bd880115e1444594881c95c55016d2 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=474d31f683724897b682ce8741ddaa19]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:17.768229 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79bd880115e1444594881c95c55016d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:17.770529 27740 tablet_service.cc:1467] Processing CreateTablet for tablet 2518578063474b668508d644a2098331 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=474d31f683724897b682ce8741ddaa19]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:17.771068 27740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2518578063474b668508d644a2098331. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:17.774480 28131 tablet_service.cc:1467] Processing CreateTablet for tablet 79bd880115e1444594881c95c55016d2 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=474d31f683724897b682ce8741ddaa19]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:17.774973 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79bd880115e1444594881c95c55016d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:17.775183 31897 tablet_bootstrap.cc:492] T 868644032be244eb89977028be916a57 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:17.776763 31897 tablet_bootstrap.cc:654] T 868644032be244eb89977028be916a57 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:17.779338 31901 tablet_bootstrap.cc:492] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:17.781323 31901 tablet_bootstrap.cc:654] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:17.787411 31898 tablet_bootstrap.cc:492] T 868644032be244eb89977028be916a57 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:17.788743 31897 tablet_bootstrap.cc:492] T 868644032be244eb89977028be916a57 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:17.788923 31897 ts_tablet_manager.cc:1367] T 868644032be244eb89977028be916a57 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.004s
I20240501 08:12:17.789801 31897 raft_consensus.cc:348] T 868644032be244eb89977028be916a57 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:17.790199 31897 raft_consensus.cc:374] T 868644032be244eb89977028be916a57 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:17.790360 31897 raft_consensus.cc:729] T 868644032be244eb89977028be916a57 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:17.790678 31897 consensus_queue.cc:260] T 868644032be244eb89977028be916a57 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:17.791152 31898 tablet_bootstrap.cc:654] T 868644032be244eb89977028be916a57 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:17.792327 31897 ts_tablet_manager.cc:1398] T 868644032be244eb89977028be916a57 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:17.792694 31897 tablet_bootstrap.cc:492] T 2518578063474b668508d644a2098331 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:17.795038 31897 tablet_bootstrap.cc:654] T 2518578063474b668508d644a2098331 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:17.796157 31906 tablet_bootstrap.cc:492] T 79bd880115e1444594881c95c55016d2 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:17.798244 31906 tablet_bootstrap.cc:654] T 79bd880115e1444594881c95c55016d2 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:17.800158 31901 tablet_bootstrap.cc:492] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:17.800432 31901 ts_tablet_manager.cc:1367] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.006s
I20240501 08:12:17.801596 31901 raft_consensus.cc:348] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:17.802000 31901 raft_consensus.cc:374] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:17.802193 31901 raft_consensus.cc:729] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:17.802526 31901 consensus_queue.cc:260] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:17.804400 31901 ts_tablet_manager.cc:1398] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 08:12:17.804782 31901 tablet_bootstrap.cc:492] T 79bd880115e1444594881c95c55016d2 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:17.806313 31900 tablet_bootstrap.cc:492] T 79bd880115e1444594881c95c55016d2 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:17.808123 31897 tablet_bootstrap.cc:492] T 2518578063474b668508d644a2098331 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:17.808326 31901 tablet_bootstrap.cc:654] T 79bd880115e1444594881c95c55016d2 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:17.808594 31897 ts_tablet_manager.cc:1367] T 2518578063474b668508d644a2098331 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.002s
I20240501 08:12:17.810992 31900 tablet_bootstrap.cc:654] T 79bd880115e1444594881c95c55016d2 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:17.811568 31899 tablet_bootstrap.cc:492] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:17.812057 31904 tablet_bootstrap.cc:492] T fc9e0f35ebcd48b5a27c57b2485e599e P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:17.811807 31897 raft_consensus.cc:348] T 2518578063474b668508d644a2098331 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:17.812340 31897 raft_consensus.cc:374] T 2518578063474b668508d644a2098331 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:17.812448 31897 raft_consensus.cc:729] T 2518578063474b668508d644a2098331 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:17.812707 31897 consensus_queue.cc:260] T 2518578063474b668508d644a2098331 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:17.814669 31897 ts_tablet_manager.cc:1398] T 2518578063474b668508d644a2098331 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:12:17.815052 31898 tablet_bootstrap.cc:492] T 868644032be244eb89977028be916a57 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:17.815251 31898 ts_tablet_manager.cc:1367] T 868644032be244eb89977028be916a57 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.000s
I20240501 08:12:17.816212 31898 raft_consensus.cc:348] T 868644032be244eb89977028be916a57 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:17.816733 31898 raft_consensus.cc:374] T 868644032be244eb89977028be916a57 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:17.816890 31898 raft_consensus.cc:729] T 868644032be244eb89977028be916a57 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:17.817229 31898 consensus_queue.cc:260] T 868644032be244eb89977028be916a57 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:17.818156 31901 tablet_bootstrap.cc:492] T 79bd880115e1444594881c95c55016d2 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:17.818392 31901 ts_tablet_manager.cc:1367] T 79bd880115e1444594881c95c55016d2 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.005s
I20240501 08:12:17.819420 31901 raft_consensus.cc:348] T 79bd880115e1444594881c95c55016d2 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:17.820600 31898 ts_tablet_manager.cc:1398] T 868644032be244eb89977028be916a57 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 08:12:17.820873 31906 tablet_bootstrap.cc:492] T 79bd880115e1444594881c95c55016d2 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:12:17.821098 31906 ts_tablet_manager.cc:1367] T 79bd880115e1444594881c95c55016d2 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.004s
I20240501 08:12:17.821274 31899 tablet_bootstrap.cc:654] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:17.822108 31906 raft_consensus.cc:348] T 79bd880115e1444594881c95c55016d2 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:17.822372 31906 raft_consensus.cc:374] T 79bd880115e1444594881c95c55016d2 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:17.822515 31906 raft_consensus.cc:729] T 79bd880115e1444594881c95c55016d2 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:12:17.822945 31898 tablet_bootstrap.cc:492] T 2518578063474b668508d644a2098331 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:17.823081 31906 consensus_queue.cc:260] T 79bd880115e1444594881c95c55016d2 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:17.824934 31906 ts_tablet_manager.cc:1398] T 79bd880115e1444594881c95c55016d2 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:17.825228 31901 raft_consensus.cc:374] T 79bd880115e1444594881c95c55016d2 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:17.825366 31901 raft_consensus.cc:729] T 79bd880115e1444594881c95c55016d2 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:17.825702 31901 consensus_queue.cc:260] T 79bd880115e1444594881c95c55016d2 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:17.825920 31904 tablet_bootstrap.cc:654] T fc9e0f35ebcd48b5a27c57b2485e599e P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:17.826478 31901 ts_tablet_manager.cc:1398] T 79bd880115e1444594881c95c55016d2 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 08:12:17.828048 31904 tablet_bootstrap.cc:492] T fc9e0f35ebcd48b5a27c57b2485e599e P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:17.828212 31904 ts_tablet_manager.cc:1367] T fc9e0f35ebcd48b5a27c57b2485e599e P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240501 08:12:17.828428 31900 tablet_bootstrap.cc:492] T 79bd880115e1444594881c95c55016d2 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:12:17.828644 31900 ts_tablet_manager.cc:1367] T 79bd880115e1444594881c95c55016d2 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.002s
I20240501 08:12:17.829608 31900 raft_consensus.cc:348] T 79bd880115e1444594881c95c55016d2 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:17.829898 31900 raft_consensus.cc:374] T 79bd880115e1444594881c95c55016d2 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:17.830080 31900 raft_consensus.cc:729] T 79bd880115e1444594881c95c55016d2 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:12:17.830581 31900 consensus_queue.cc:260] T 79bd880115e1444594881c95c55016d2 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:17.831632 31900 ts_tablet_manager.cc:1398] T 79bd880115e1444594881c95c55016d2 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:17.831566 31904 raft_consensus.cc:348] T fc9e0f35ebcd48b5a27c57b2485e599e P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:17.831930 31904 raft_consensus.cc:374] T fc9e0f35ebcd48b5a27c57b2485e599e P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:17.832077 31904 raft_consensus.cc:729] T fc9e0f35ebcd48b5a27c57b2485e599e P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:17.834782 31904 consensus_queue.cc:260] T fc9e0f35ebcd48b5a27c57b2485e599e P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:17.836088 31898 tablet_bootstrap.cc:654] T 2518578063474b668508d644a2098331 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:17.839618 31898 tablet_bootstrap.cc:492] T 2518578063474b668508d644a2098331 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:17.839807 31898 ts_tablet_manager.cc:1367] T 2518578063474b668508d644a2098331 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.001s
I20240501 08:12:17.840579 31898 raft_consensus.cc:348] T 2518578063474b668508d644a2098331 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:17.840792 31898 raft_consensus.cc:374] T 2518578063474b668508d644a2098331 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:17.840893 31898 raft_consensus.cc:729] T 2518578063474b668508d644a2098331 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:17.841006 31904 ts_tablet_manager.cc:1398] T fc9e0f35ebcd48b5a27c57b2485e599e P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240501 08:12:17.841177 31898 consensus_queue.cc:260] T 2518578063474b668508d644a2098331 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:17.841874 31898 ts_tablet_manager.cc:1398] T 2518578063474b668508d644a2098331 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:17.843849 31899 tablet_bootstrap.cc:492] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:12:17.844024 31899 ts_tablet_manager.cc:1367] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.000s
I20240501 08:12:17.844720 31899 raft_consensus.cc:348] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:17.844889 31899 raft_consensus.cc:374] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:17.845548 31899 raft_consensus.cc:729] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:12:17.845808 31899 consensus_queue.cc:260] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:17.848094 31899 ts_tablet_manager.cc:1398] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240501 08:12:17.849833 31899 tablet_bootstrap.cc:492] T fc9e0f35ebcd48b5a27c57b2485e599e P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:17.851452 31899 tablet_bootstrap.cc:654] T fc9e0f35ebcd48b5a27c57b2485e599e P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:17.855852 31499 raft_consensus.cc:482] T 79bd880115e1444594881c95c55016d2 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:17.856032 31499 raft_consensus.cc:504] T 79bd880115e1444594881c95c55016d2 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:17.856729 31499 leader_election.cc:288] T 79bd880115e1444594881c95c55016d2 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:17.857280 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bd880115e1444594881c95c55016d2" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:12:17.857503 28022 raft_consensus.cc:2454] T 79bd880115e1444594881c95c55016d2 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:12:17.858008 28087 leader_election.cc:302] T 79bd880115e1444594881c95c55016d2 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:12:17.857959 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bd880115e1444594881c95c55016d2" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:17.858208 30804 raft_consensus.cc:2454] T 79bd880115e1444594881c95c55016d2 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:12:17.858498 31499 raft_consensus.cc:2781] T 79bd880115e1444594881c95c55016d2 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:17.858625 31499 raft_consensus.cc:482] T 79bd880115e1444594881c95c55016d2 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:17.858719 31499 raft_consensus.cc:3037] T 79bd880115e1444594881c95c55016d2 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:17.859505 31899 tablet_bootstrap.cc:492] T fc9e0f35ebcd48b5a27c57b2485e599e P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:12:17.859674 31899 ts_tablet_manager.cc:1367] T fc9e0f35ebcd48b5a27c57b2485e599e P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240501 08:12:17.860293 31899 raft_consensus.cc:348] T fc9e0f35ebcd48b5a27c57b2485e599e P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:17.860514 31899 raft_consensus.cc:374] T fc9e0f35ebcd48b5a27c57b2485e599e P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:17.860589 31899 raft_consensus.cc:729] T fc9e0f35ebcd48b5a27c57b2485e599e P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:12:17.860859 31899 consensus_queue.cc:260] T fc9e0f35ebcd48b5a27c57b2485e599e P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:17.860883 31499 raft_consensus.cc:504] T 79bd880115e1444594881c95c55016d2 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:17.861371 31899 ts_tablet_manager.cc:1398] T fc9e0f35ebcd48b5a27c57b2485e599e P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:17.861701 31899 tablet_bootstrap.cc:492] T 2518578063474b668508d644a2098331 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:17.862423 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bd880115e1444594881c95c55016d2" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:12:17.862416 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bd880115e1444594881c95c55016d2" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:17.862622 30804 raft_consensus.cc:3037] T 79bd880115e1444594881c95c55016d2 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:17.862648 28022 raft_consensus.cc:3037] T 79bd880115e1444594881c95c55016d2 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:17.864531 31899 tablet_bootstrap.cc:654] T 2518578063474b668508d644a2098331 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:17.865514 30804 raft_consensus.cc:2454] T 79bd880115e1444594881c95c55016d2 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:12:17.865500 28022 raft_consensus.cc:2454] T 79bd880115e1444594881c95c55016d2 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:12:17.866288 28086 leader_election.cc:302] T 79bd880115e1444594881c95c55016d2 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:12:17.866636 31896 raft_consensus.cc:2781] T 79bd880115e1444594881c95c55016d2 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:17.866667 31899 tablet_bootstrap.cc:492] T 2518578063474b668508d644a2098331 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:12:17.866786 31896 raft_consensus.cc:686] T 79bd880115e1444594881c95c55016d2 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:12:17.867117 31896 consensus_queue.cc:237] T 79bd880115e1444594881c95c55016d2 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:17.867940 31499 leader_election.cc:288] T 79bd880115e1444594881c95c55016d2 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:17.868641 31899 ts_tablet_manager.cc:1367] T 2518578063474b668508d644a2098331 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 08:12:17.869506 31899 raft_consensus.cc:348] T 2518578063474b668508d644a2098331 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:17.869758 31899 raft_consensus.cc:374] T 2518578063474b668508d644a2098331 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:17.869916 31899 raft_consensus.cc:729] T 2518578063474b668508d644a2098331 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:12:17.870252 31899 consensus_queue.cc:260] T 2518578063474b668508d644a2098331 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:17.870771 31899 ts_tablet_manager.cc:1398] T 2518578063474b668508d644a2098331 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:17.871176 27613 catalog_manager.cc:5484] T 79bd880115e1444594881c95c55016d2 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:17.881415 31791 raft_consensus.cc:482] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:17.881592 31791 raft_consensus.cc:504] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:17.882249 31791 leader_election.cc:288] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:17.882730 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "868644032be244eb89977028be916a57" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:17.882748 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "868644032be244eb89977028be916a57" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:17.883013 30936 raft_consensus.cc:2454] T 868644032be244eb89977028be916a57 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:12:17.883028 30537 raft_consensus.cc:2454] T 868644032be244eb89977028be916a57 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:12:17.884027 27697 leader_election.cc:302] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 6331f77dae9d452184e8cd0cbf5f1b43; no voters:
I20240501 08:12:17.884357 31791 raft_consensus.cc:2781] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:17.884479 31791 raft_consensus.cc:482] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:17.884570 31791 raft_consensus.cc:3037] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:17.886664 31791 raft_consensus.cc:504] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:17.887302 31791 leader_election.cc:288] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Requested vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:17.887857 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "868644032be244eb89977028be916a57" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:17.887876 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "868644032be244eb89977028be916a57" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:17.888088 30936 raft_consensus.cc:3037] T 868644032be244eb89977028be916a57 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:17.888078 30537 raft_consensus.cc:3037] T 868644032be244eb89977028be916a57 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:17.891045 30936 raft_consensus.cc:2454] T 868644032be244eb89977028be916a57 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:12:17.891045 30537 raft_consensus.cc:2454] T 868644032be244eb89977028be916a57 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:12:17.891817 27696 leader_election.cc:302] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:17.892678 31791 raft_consensus.cc:2781] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:17.892935 31791 raft_consensus.cc:686] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:12:17.894423 31791 consensus_queue.cc:237] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:17.898610 27613 catalog_manager.cc:5484] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:17.997284 31872 raft_consensus.cc:482] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:17.997491 31872 raft_consensus.cc:504] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:17.998268 31872 leader_election.cc:288] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:17.998809 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc9e0f35ebcd48b5a27c57b2485e599e" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:17.998833 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc9e0f35ebcd48b5a27c57b2485e599e" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:12:17.999019 30672 raft_consensus.cc:2454] T fc9e0f35ebcd48b5a27c57b2485e599e P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:17.999049 27759 raft_consensus.cc:2454] T fc9e0f35ebcd48b5a27c57b2485e599e P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:17.999504 30739 leader_election.cc:302] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 64998323a1f4482d8704bf20594d13a5; no voters:
I20240501 08:12:17.999787 31872 raft_consensus.cc:2781] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:17.999917 31872 raft_consensus.cc:482] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:17.999991 31872 raft_consensus.cc:3037] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.002067 31872 raft_consensus.cc:504] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:18.002666 31872 leader_election.cc:288] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Requested vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:18.003311 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc9e0f35ebcd48b5a27c57b2485e599e" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:18.003311 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc9e0f35ebcd48b5a27c57b2485e599e" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:12:18.003540 27759 raft_consensus.cc:3037] T fc9e0f35ebcd48b5a27c57b2485e599e P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.003541 30672 raft_consensus.cc:3037] T fc9e0f35ebcd48b5a27c57b2485e599e P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.006542 30672 raft_consensus.cc:2454] T fc9e0f35ebcd48b5a27c57b2485e599e P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:18.006542 27759 raft_consensus.cc:2454] T fc9e0f35ebcd48b5a27c57b2485e599e P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:18.007012 30737 leader_election.cc:302] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:18.007436 31872 raft_consensus.cc:2781] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:18.007573 31872 raft_consensus.cc:686] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Becoming Leader. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Running, Role: LEADER
I20240501 08:12:18.007849 31872 consensus_queue.cc:237] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:18.008200 27613 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:18.013808 27620 catalog_manager.cc:5484] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64998323a1f4482d8704bf20594d13a5 (127.21.84.135). New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:18.029279 30537 raft_consensus.cc:1261] T 868644032be244eb89977028be916a57 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:18.029238 30936 raft_consensus.cc:1261] T 868644032be244eb89977028be916a57 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:18.029912 31791 consensus_queue.cc:1035] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:18.030213 31789 consensus_queue.cc:1035] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:18.034857 27620 catalog_manager.cc:5576] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845) does not have the latest schema for tablet 868644032be244eb89977028be916a57 (table rebalancer_dynamic_table [id=3a634adcce0b4a47a32f2e24acbf09ed]). Expected version 1 got 0
I20240501 08:12:18.037685 31926 tablet.cc:1722] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:18.038565 31926 tablet.cc:1620] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43: MemRowSet was empty: no flush needed.
I20240501 08:12:18.039851 27620 catalog_manager.cc:5576] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293) does not have the latest schema for tablet 868644032be244eb89977028be916a57 (table rebalancer_dynamic_table [id=3a634adcce0b4a47a32f2e24acbf09ed]). Expected version 1 got 0
I20240501 08:12:18.049734 27613 catalog_manager.cc:5576] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293) does not have the latest schema for tablet 868644032be244eb89977028be916a57 (table rebalancer_dynamic_table [id=3a634adcce0b4a47a32f2e24acbf09ed]). Expected version 1 got 0
I20240501 08:12:18.055493 31927 tablet.cc:1722] T 868644032be244eb89977028be916a57 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:18.056138 31927 tablet.cc:1620] T 868644032be244eb89977028be916a57 P a29cd590999b4f32849ff5689cd53c11: MemRowSet was empty: no flush needed.
I20240501 08:12:18.057385 31928 tablet.cc:1722] T 868644032be244eb89977028be916a57 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:18.057607 27620 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=3a634adcce0b4a47a32f2e24acbf09ed] alter complete (version 1)
I20240501 08:12:18.058005 31928 tablet.cc:1620] T 868644032be244eb89977028be916a57 P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:18.058005 27617 catalog_manager.cc:5576] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845) does not have the latest schema for tablet 868644032be244eb89977028be916a57 (table rebalancer_dynamic_table [id=3a634adcce0b4a47a32f2e24acbf09ed]). Expected version 1 got 0
I20240501 08:12:18.101104 31789 raft_consensus.cc:482] T 2518578063474b668508d644a2098331 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:18.101349 31789 raft_consensus.cc:504] T 2518578063474b668508d644a2098331 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:18.102137 31789 leader_election.cc:288] T 2518578063474b668508d644a2098331 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:18.102795 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2518578063474b668508d644a2098331" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:18.103075 30537 raft_consensus.cc:2454] T 2518578063474b668508d644a2098331 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:12:18.103487 27697 leader_election.cc:302] T 2518578063474b668508d644a2098331 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 6331f77dae9d452184e8cd0cbf5f1b43; no voters:
I20240501 08:12:18.103053 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2518578063474b668508d644a2098331" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:18.103781 31789 raft_consensus.cc:2781] T 2518578063474b668508d644a2098331 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:18.103880 31789 raft_consensus.cc:482] T 2518578063474b668508d644a2098331 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:18.103853 30936 raft_consensus.cc:2454] T 2518578063474b668508d644a2098331 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:12:18.103977 31789 raft_consensus.cc:3037] T 2518578063474b668508d644a2098331 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.106685 31789 raft_consensus.cc:504] T 2518578063474b668508d644a2098331 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:18.107527 31789 leader_election.cc:288] T 2518578063474b668508d644a2098331 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Requested vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:18.108304 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2518578063474b668508d644a2098331" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:18.108371 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2518578063474b668508d644a2098331" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:18.108521 30537 raft_consensus.cc:3037] T 2518578063474b668508d644a2098331 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.108718 30936 raft_consensus.cc:3037] T 2518578063474b668508d644a2098331 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.111557 30936 raft_consensus.cc:2454] T 2518578063474b668508d644a2098331 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:12:18.111557 30537 raft_consensus.cc:2454] T 2518578063474b668508d644a2098331 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:12:18.112011 27697 leader_election.cc:302] T 2518578063474b668508d644a2098331 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 6331f77dae9d452184e8cd0cbf5f1b43; no voters:
I20240501 08:12:18.112488 31789 raft_consensus.cc:2781] T 2518578063474b668508d644a2098331 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:18.112666 31789 raft_consensus.cc:686] T 2518578063474b668508d644a2098331 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:12:18.112917 31789 consensus_queue.cc:237] T 2518578063474b668508d644a2098331 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:18.116789 27617 catalog_manager.cc:5484] T 2518578063474b668508d644a2098331 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:18.168282 27617 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:18.175905 31926 tablet.cc:1722] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:18.176476 31926 tablet.cc:1620] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43: MemRowSet was empty: no flush needed.
I20240501 08:12:18.176851 31927 tablet.cc:1722] T 868644032be244eb89977028be916a57 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:18.177342 31927 tablet.cc:1620] T 868644032be244eb89977028be916a57 P a29cd590999b4f32849ff5689cd53c11: MemRowSet was empty: no flush needed.
I20240501 08:12:18.182003 31928 tablet.cc:1722] T 868644032be244eb89977028be916a57 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:18.182523 31928 tablet.cc:1620] T 868644032be244eb89977028be916a57 P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:18.187899 27617 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=3a634adcce0b4a47a32f2e24acbf09ed] alter complete (version 2)
I20240501 08:12:18.248838 31895 consensus_queue.cc:1035] T 79bd880115e1444594881c95c55016d2 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:18.255376 31896 consensus_queue.cc:1035] T 79bd880115e1444594881c95c55016d2 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:18.292834 27611 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:18.299954 31926 tablet.cc:1722] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:18.301190 31927 tablet.cc:1722] T 868644032be244eb89977028be916a57 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:18.301460 31928 tablet.cc:1722] T 868644032be244eb89977028be916a57 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:18.312296 27611 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table_new_name [id=3a634adcce0b4a47a32f2e24acbf09ed] alter complete (version 3)
I20240501 08:12:18.414894 27613 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:18.415134 27613 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:18.418275 27613 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 868644032be244eb89977028be916a57
I20240501 08:12:18.419086 30514 tablet_service.cc:1514] Processing DeleteTablet for tablet 868644032be244eb89977028be916a57 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:18 UTC) from {username='slave'} at 127.0.0.1:45078
I20240501 08:12:18.419399 30916 tablet_service.cc:1514] Processing DeleteTablet for tablet 868644032be244eb89977028be916a57 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:18 UTC) from {username='slave'} at 127.0.0.1:54036
I20240501 08:12:18.419541 27741 tablet_service.cc:1514] Processing DeleteTablet for tablet 868644032be244eb89977028be916a57 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:18 UTC) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:18.419867 31938 tablet_replica.cc:330] T 868644032be244eb89977028be916a57 P 31811708c6774037b82df44024b229b6: stopping tablet replica
I20240501 08:12:18.420037 31939 tablet_replica.cc:330] T 868644032be244eb89977028be916a57 P a29cd590999b4f32849ff5689cd53c11: stopping tablet replica
I20240501 08:12:18.420114 31938 raft_consensus.cc:2229] T 868644032be244eb89977028be916a57 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:18.420347 31938 raft_consensus.cc:2258] T 868644032be244eb89977028be916a57 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:18.420470 31939 raft_consensus.cc:2229] T 868644032be244eb89977028be916a57 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:18.420907 31939 raft_consensus.cc:2258] T 868644032be244eb89977028be916a57 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:18.421442 31940 tablet_replica.cc:330] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:18.421648 31940 raft_consensus.cc:2229] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:18.422039 31940 raft_consensus.cc:2258] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:18.422024 31938 ts_tablet_manager.cc:1875] T 868644032be244eb89977028be916a57 P 31811708c6774037b82df44024b229b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:18.422721 31940 ts_tablet_manager.cc:1875] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:18.425288 31939 ts_tablet_manager.cc:1875] T 868644032be244eb89977028be916a57 P a29cd590999b4f32849ff5689cd53c11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:18.430724 31940 ts_tablet_manager.cc:1888] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:18.430866 31940 log.cc:1196] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/868644032be244eb89977028be916a57
I20240501 08:12:18.430852 31938 ts_tablet_manager.cc:1888] T 868644032be244eb89977028be916a57 P 31811708c6774037b82df44024b229b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:18.430985 31938 log.cc:1196] T 868644032be244eb89977028be916a57 P 31811708c6774037b82df44024b229b6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/wals/868644032be244eb89977028be916a57
I20240501 08:12:18.431412 31940 ts_tablet_manager.cc:1909] T 868644032be244eb89977028be916a57 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting consensus metadata
I20240501 08:12:18.431495 31938 ts_tablet_manager.cc:1909] T 868644032be244eb89977028be916a57 P 31811708c6774037b82df44024b229b6: Deleting consensus metadata
I20240501 08:12:18.432446 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet 868644032be244eb89977028be916a57 (table rebalancer_dynamic_table_new_name [id=3a634adcce0b4a47a32f2e24acbf09ed]) successfully deleted
I20240501 08:12:18.432840 27607 catalog_manager.cc:4830] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): tablet 868644032be244eb89977028be916a57 (table rebalancer_dynamic_table_new_name [id=3a634adcce0b4a47a32f2e24acbf09ed]) successfully deleted
I20240501 08:12:18.433408 31939 ts_tablet_manager.cc:1888] T 868644032be244eb89977028be916a57 P a29cd590999b4f32849ff5689cd53c11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:18.433517 31939 log.cc:1196] T 868644032be244eb89977028be916a57 P a29cd590999b4f32849ff5689cd53c11: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/wals/868644032be244eb89977028be916a57
I20240501 08:12:18.433966 31939 ts_tablet_manager.cc:1909] T 868644032be244eb89977028be916a57 P a29cd590999b4f32849ff5689cd53c11: Deleting consensus metadata
I20240501 08:12:18.435067 27606 catalog_manager.cc:4830] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): tablet 868644032be244eb89977028be916a57 (table rebalancer_dynamic_table_new_name [id=3a634adcce0b4a47a32f2e24acbf09ed]) successfully deleted
I20240501 08:12:18.468318 31748 consensus_queue.cc:1035] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:18.478884 31743 consensus_queue.cc:1035] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:18.520747 27613 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:18.527854 30651 tablet_service.cc:1467] Processing CreateTablet for tablet 0d6978a9bb3b432db852d43bd58838e6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c1c4949221c344a1adaef209ccd26c0a]), partition=
I20240501 08:12:18.528131 30916 tablet_service.cc:1467] Processing CreateTablet for tablet 0d6978a9bb3b432db852d43bd58838e6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c1c4949221c344a1adaef209ccd26c0a]), partition=
I20240501 08:12:18.528287 30651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d6978a9bb3b432db852d43bd58838e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:18.528342 30514 tablet_service.cc:1467] Processing CreateTablet for tablet 0d6978a9bb3b432db852d43bd58838e6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c1c4949221c344a1adaef209ccd26c0a]), partition=
I20240501 08:12:18.528496 30916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d6978a9bb3b432db852d43bd58838e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:18.528704 30514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d6978a9bb3b432db852d43bd58838e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:18.536948 31946 tablet_bootstrap.cc:492] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:18.537612 31945 tablet_bootstrap.cc:492] T 0d6978a9bb3b432db852d43bd58838e6 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:18.539486 31945 tablet_bootstrap.cc:654] T 0d6978a9bb3b432db852d43bd58838e6 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:18.539696 31947 tablet_bootstrap.cc:492] T 0d6978a9bb3b432db852d43bd58838e6 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:18.540352 31946 tablet_bootstrap.cc:654] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:18.541141 31947 tablet_bootstrap.cc:654] T 0d6978a9bb3b432db852d43bd58838e6 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:18.548105 31945 tablet_bootstrap.cc:492] T 0d6978a9bb3b432db852d43bd58838e6 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:18.548364 31945 ts_tablet_manager.cc:1367] T 0d6978a9bb3b432db852d43bd58838e6 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20240501 08:12:18.549181 31945 raft_consensus.cc:348] T 0d6978a9bb3b432db852d43bd58838e6 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:18.549437 31945 raft_consensus.cc:374] T 0d6978a9bb3b432db852d43bd58838e6 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:18.549521 31945 raft_consensus.cc:729] T 0d6978a9bb3b432db852d43bd58838e6 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:18.549741 31945 consensus_queue.cc:260] T 0d6978a9bb3b432db852d43bd58838e6 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:18.550366 31945 ts_tablet_manager.cc:1398] T 0d6978a9bb3b432db852d43bd58838e6 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:18.551564 31946 tablet_bootstrap.cc:492] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:18.552018 31946 ts_tablet_manager.cc:1367] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.006s
I20240501 08:12:18.552855 31946 raft_consensus.cc:348] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:18.553133 31946 raft_consensus.cc:374] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:18.553253 31946 raft_consensus.cc:729] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:18.553543 31946 consensus_queue.cc:260] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:18.555426 31946 ts_tablet_manager.cc:1398] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 08:12:18.557180 31947 tablet_bootstrap.cc:492] T 0d6978a9bb3b432db852d43bd58838e6 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:18.557400 31947 ts_tablet_manager.cc:1367] T 0d6978a9bb3b432db852d43bd58838e6 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.012s
I20240501 08:12:18.559484 31947 raft_consensus.cc:348] T 0d6978a9bb3b432db852d43bd58838e6 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:18.559739 31947 raft_consensus.cc:374] T 0d6978a9bb3b432db852d43bd58838e6 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:18.559859 31947 raft_consensus.cc:729] T 0d6978a9bb3b432db852d43bd58838e6 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:18.560186 31947 consensus_queue.cc:260] T 0d6978a9bb3b432db852d43bd58838e6 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:18.560925 31947 ts_tablet_manager.cc:1398] T 0d6978a9bb3b432db852d43bd58838e6 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:18.619949 31931 consensus_queue.cc:1035] T 2518578063474b668508d644a2098331 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:18.624753 31791 consensus_queue.cc:1035] T 2518578063474b668508d644a2098331 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:18.690690 27613 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_extra_table_30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:18.700495 30651 tablet_service.cc:1467] Processing CreateTablet for tablet 4fd595c79cdc43f9a8afadb6b548bf64 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=9da53f67716d4d2892b2ff2b43922ec5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:18.701041 30651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fd595c79cdc43f9a8afadb6b548bf64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:18.701246 30916 tablet_service.cc:1467] Processing CreateTablet for tablet 4fd595c79cdc43f9a8afadb6b548bf64 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=9da53f67716d4d2892b2ff2b43922ec5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:18.701699 30916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fd595c79cdc43f9a8afadb6b548bf64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:18.702365 27872 tablet_service.cc:1467] Processing CreateTablet for tablet 4fd595c79cdc43f9a8afadb6b548bf64 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=9da53f67716d4d2892b2ff2b43922ec5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:18.702775 27872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fd595c79cdc43f9a8afadb6b548bf64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:18.706387 30784 tablet_service.cc:1467] Processing CreateTablet for tablet 73f40a7ef6954b34a48d8c5aeeaebc0a (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=9da53f67716d4d2892b2ff2b43922ec5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:18.706813 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73f40a7ef6954b34a48d8c5aeeaebc0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:18.708835 31947 tablet_bootstrap.cc:492] T 4fd595c79cdc43f9a8afadb6b548bf64 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:18.711839 31947 tablet_bootstrap.cc:654] T 4fd595c79cdc43f9a8afadb6b548bf64 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:18.713819 30514 tablet_service.cc:1467] Processing CreateTablet for tablet 73f40a7ef6954b34a48d8c5aeeaebc0a (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=9da53f67716d4d2892b2ff2b43922ec5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:18.714314 30514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73f40a7ef6954b34a48d8c5aeeaebc0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:18.714591 31955 tablet_bootstrap.cc:492] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:18.715687 28000 tablet_service.cc:1467] Processing CreateTablet for tablet 38c47b76fb304221b60d2b572485ad9f (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=9da53f67716d4d2892b2ff2b43922ec5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:18.716238 28000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38c47b76fb304221b60d2b572485ad9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:18.717456 31945 tablet_bootstrap.cc:492] T 4fd595c79cdc43f9a8afadb6b548bf64 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:18.719782 31945 tablet_bootstrap.cc:654] T 4fd595c79cdc43f9a8afadb6b548bf64 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:18.721185 28002 tablet_service.cc:1467] Processing CreateTablet for tablet 73f40a7ef6954b34a48d8c5aeeaebc0a (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=9da53f67716d4d2892b2ff2b43922ec5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:18.721648 28002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73f40a7ef6954b34a48d8c5aeeaebc0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:18.725741 30517 tablet_service.cc:1467] Processing CreateTablet for tablet 38c47b76fb304221b60d2b572485ad9f (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=9da53f67716d4d2892b2ff2b43922ec5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:18.726224 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38c47b76fb304221b60d2b572485ad9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:18.727488 31953 tablet_bootstrap.cc:492] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:18.729197 31953 tablet_bootstrap.cc:654] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:18.735844 31946 tablet_bootstrap.cc:492] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:18.736634 30916 tablet_service.cc:1467] Processing CreateTablet for tablet 38c47b76fb304221b60d2b572485ad9f (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=9da53f67716d4d2892b2ff2b43922ec5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:18.737625 31946 tablet_bootstrap.cc:654] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:18.737076 30916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38c47b76fb304221b60d2b572485ad9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:18.741616 31955 tablet_bootstrap.cc:654] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:18.743125 31958 tablet_bootstrap.cc:492] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:18.745765 31958 tablet_bootstrap.cc:654] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:18.750618 31947 tablet_bootstrap.cc:492] T 4fd595c79cdc43f9a8afadb6b548bf64 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:18.750977 31947 ts_tablet_manager.cc:1367] T 4fd595c79cdc43f9a8afadb6b548bf64 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.002s
I20240501 08:12:18.751976 31947 raft_consensus.cc:348] T 4fd595c79cdc43f9a8afadb6b548bf64 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:18.752355 31947 raft_consensus.cc:374] T 4fd595c79cdc43f9a8afadb6b548bf64 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:18.752519 31947 raft_consensus.cc:729] T 4fd595c79cdc43f9a8afadb6b548bf64 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:18.753082 31945 tablet_bootstrap.cc:492] T 4fd595c79cdc43f9a8afadb6b548bf64 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:18.752874 31947 consensus_queue.cc:260] T 4fd595c79cdc43f9a8afadb6b548bf64 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:18.753412 31945 ts_tablet_manager.cc:1367] T 4fd595c79cdc43f9a8afadb6b548bf64 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.004s
I20240501 08:12:18.754489 31945 raft_consensus.cc:348] T 4fd595c79cdc43f9a8afadb6b548bf64 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:18.754863 31945 raft_consensus.cc:374] T 4fd595c79cdc43f9a8afadb6b548bf64 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:18.754998 31945 raft_consensus.cc:729] T 4fd595c79cdc43f9a8afadb6b548bf64 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:18.755369 31945 consensus_queue.cc:260] T 4fd595c79cdc43f9a8afadb6b548bf64 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:18.757248 31945 ts_tablet_manager.cc:1398] T 4fd595c79cdc43f9a8afadb6b548bf64 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:18.759974 31958 tablet_bootstrap.cc:492] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:12:18.760690 31958 ts_tablet_manager.cc:1367] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20240501 08:12:18.761380 31953 tablet_bootstrap.cc:492] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:18.761477 31955 tablet_bootstrap.cc:492] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:12:18.761693 31955 ts_tablet_manager.cc:1367] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.008s
I20240501 08:12:18.761685 31953 ts_tablet_manager.cc:1367] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.006s
I20240501 08:12:18.761751 31958 raft_consensus.cc:348] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:18.762537 31955 raft_consensus.cc:348] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:18.762766 31955 raft_consensus.cc:374] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:18.762813 31958 raft_consensus.cc:374] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:18.762889 31955 raft_consensus.cc:729] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:12:18.763195 31955 consensus_queue.cc:260] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:18.763024 31958 raft_consensus.cc:729] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:12:18.763907 31955 ts_tablet_manager.cc:1398] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:12:18.764204 31958 consensus_queue.cc:260] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:18.765594 31947 ts_tablet_manager.cc:1398] T 4fd595c79cdc43f9a8afadb6b548bf64 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20240501 08:12:18.765674 31958 ts_tablet_manager.cc:1398] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:12:18.766499 31958 tablet_bootstrap.cc:492] T 38c47b76fb304221b60d2b572485ad9f P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:18.769073 31953 raft_consensus.cc:348] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:18.769341 31953 raft_consensus.cc:374] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:18.769430 31953 raft_consensus.cc:729] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:18.769917 31953 consensus_queue.cc:260] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:18.769906 31947 tablet_bootstrap.cc:492] T 38c47b76fb304221b60d2b572485ad9f P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:18.771731 31953 ts_tablet_manager.cc:1398] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240501 08:12:18.773262 31958 tablet_bootstrap.cc:654] T 38c47b76fb304221b60d2b572485ad9f P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:18.778218 31947 tablet_bootstrap.cc:654] T 38c47b76fb304221b60d2b572485ad9f P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:18.781230 31947 tablet_bootstrap.cc:492] T 38c47b76fb304221b60d2b572485ad9f P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:18.781402 31947 ts_tablet_manager.cc:1367] T 38c47b76fb304221b60d2b572485ad9f P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240501 08:12:18.782112 31947 raft_consensus.cc:348] T 38c47b76fb304221b60d2b572485ad9f P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:18.782524 31947 raft_consensus.cc:374] T 38c47b76fb304221b60d2b572485ad9f P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:18.782685 31947 raft_consensus.cc:729] T 38c47b76fb304221b60d2b572485ad9f P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:18.783083 31958 tablet_bootstrap.cc:492] T 38c47b76fb304221b60d2b572485ad9f P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:12:18.783269 31958 ts_tablet_manager.cc:1367] T 38c47b76fb304221b60d2b572485ad9f P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.003s
I20240501 08:12:18.783136 31947 consensus_queue.cc:260] T 38c47b76fb304221b60d2b572485ad9f P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:18.784098 31958 raft_consensus.cc:348] T 38c47b76fb304221b60d2b572485ad9f P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:18.784298 31958 raft_consensus.cc:374] T 38c47b76fb304221b60d2b572485ad9f P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:18.784377 31958 raft_consensus.cc:729] T 38c47b76fb304221b60d2b572485ad9f P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:12:18.784632 31946 tablet_bootstrap.cc:492] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:18.784623 31958 consensus_queue.cc:260] T 38c47b76fb304221b60d2b572485ad9f P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:18.784823 31946 ts_tablet_manager.cc:1367] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.049s user 0.009s sys 0.006s
I20240501 08:12:18.785154 31958 ts_tablet_manager.cc:1398] T 38c47b76fb304221b60d2b572485ad9f P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:18.785756 31946 raft_consensus.cc:348] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:18.786401 31946 raft_consensus.cc:374] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:18.786535 31946 raft_consensus.cc:729] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:18.786825 31946 consensus_queue.cc:260] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:18.787475 31946 ts_tablet_manager.cc:1398] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:18.788030 31946 tablet_bootstrap.cc:492] T 38c47b76fb304221b60d2b572485ad9f P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:18.788718 31947 ts_tablet_manager.cc:1398] T 38c47b76fb304221b60d2b572485ad9f P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240501 08:12:18.791797 31946 tablet_bootstrap.cc:654] T 38c47b76fb304221b60d2b572485ad9f P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:18.795821 31946 tablet_bootstrap.cc:492] T 38c47b76fb304221b60d2b572485ad9f P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:18.796053 31946 ts_tablet_manager.cc:1367] T 38c47b76fb304221b60d2b572485ad9f P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s
I20240501 08:12:18.796726 31946 raft_consensus.cc:348] T 38c47b76fb304221b60d2b572485ad9f P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:18.797101 31946 raft_consensus.cc:374] T 38c47b76fb304221b60d2b572485ad9f P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:18.797255 31946 raft_consensus.cc:729] T 38c47b76fb304221b60d2b572485ad9f P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:18.797531 31946 consensus_queue.cc:260] T 38c47b76fb304221b60d2b572485ad9f P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:18.798235 31946 ts_tablet_manager.cc:1398] T 38c47b76fb304221b60d2b572485ad9f P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:18.820921 31918 raft_consensus.cc:482] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:18.821143 31918 raft_consensus.cc:504] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:18.821877 31918 leader_election.cc:288] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:18.822454 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6978a9bb3b432db852d43bd58838e6" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:18.822450 30671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6978a9bb3b432db852d43bd58838e6" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:18.822683 30671 raft_consensus.cc:2454] T 0d6978a9bb3b432db852d43bd58838e6 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 0.
I20240501 08:12:18.822728 30936 raft_consensus.cc:2454] T 0d6978a9bb3b432db852d43bd58838e6 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 0.
I20240501 08:12:18.823119 30470 leader_election.cc:302] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:18.823377 31918 raft_consensus.cc:2781] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:18.823474 31918 raft_consensus.cc:482] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:18.823571 31918 raft_consensus.cc:3037] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.826050 31918 raft_consensus.cc:504] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:18.827024 31918 leader_election.cc:288] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 election: Requested vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:18.827499 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6978a9bb3b432db852d43bd58838e6" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:18.827540 30671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6978a9bb3b432db852d43bd58838e6" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:18.827718 30671 raft_consensus.cc:3037] T 0d6978a9bb3b432db852d43bd58838e6 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.827736 30936 raft_consensus.cc:3037] T 0d6978a9bb3b432db852d43bd58838e6 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.829160 31869 raft_consensus.cc:482] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:18.829355 31869 raft_consensus.cc:504] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:18.829824 30936 raft_consensus.cc:2454] T 0d6978a9bb3b432db852d43bd58838e6 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 1.
I20240501 08:12:18.830298 31869 leader_election.cc:288] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:18.830809 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f40a7ef6954b34a48d8c5aeeaebc0a" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:18.830832 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f40a7ef6954b34a48d8c5aeeaebc0a" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:18.831034 30537 raft_consensus.cc:2454] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 0.
I20240501 08:12:18.831068 30803 raft_consensus.cc:2454] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 0.
I20240501 08:12:18.831503 27958 leader_election.cc:302] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:12:18.831763 31869 raft_consensus.cc:2781] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:18.831810 30472 leader_election.cc:302] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:18.831879 31869 raft_consensus.cc:482] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:18.831964 31869 raft_consensus.cc:3037] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.832134 31918 raft_consensus.cc:2781] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:18.832235 30671 raft_consensus.cc:2454] T 0d6978a9bb3b432db852d43bd58838e6 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 1.
I20240501 08:12:18.832751 31918 raft_consensus.cc:686] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Becoming Leader. State: Replica: 31811708c6774037b82df44024b229b6, State: Running, Role: LEADER
I20240501 08:12:18.833068 31918 consensus_queue.cc:237] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:18.834803 31869 raft_consensus.cc:504] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:18.835708 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f40a7ef6954b34a48d8c5aeeaebc0a" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:18.835918 31869 leader_election.cc:288] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Requested vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:18.835958 30803 raft_consensus.cc:3037] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.838341 30803 raft_consensus.cc:2454] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 1.
I20240501 08:12:18.838698 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f40a7ef6954b34a48d8c5aeeaebc0a" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:18.838714 27956 leader_election.cc:302] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:12:18.838914 30537 raft_consensus.cc:3037] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.839025 31869 raft_consensus.cc:2781] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:18.839428 31869 raft_consensus.cc:686] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Becoming Leader. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Running, Role: LEADER
I20240501 08:12:18.839804 31869 consensus_queue.cc:237] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:18.842115 30537 raft_consensus.cc:2454] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 1.
I20240501 08:12:18.843801 27617 catalog_manager.cc:5484] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to aad81047009a4ffcbc450b60030583b5 (127.21.84.131). New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:18.843793 27620 catalog_manager.cc:5484] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31811708c6774037b82df44024b229b6 (127.21.84.133). New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:18.928843 31842 raft_consensus.cc:482] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:18.929005 31842 raft_consensus.cc:504] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:18.929746 31842 leader_election.cc:288] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:18.930280 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fd595c79cdc43f9a8afadb6b548bf64" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:18.930516 30936 raft_consensus.cc:2454] T 4fd595c79cdc43f9a8afadb6b548bf64 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:12:18.931097 30671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fd595c79cdc43f9a8afadb6b548bf64" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:18.931245 27827 leader_election.cc:302] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a29cd590999b4f32849ff5689cd53c11, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:18.931360 30671 raft_consensus.cc:2454] T 4fd595c79cdc43f9a8afadb6b548bf64 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:12:18.931530 31842 raft_consensus.cc:2781] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:18.931646 31842 raft_consensus.cc:482] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:18.931722 31842 raft_consensus.cc:3037] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.934701 31842 raft_consensus.cc:504] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:18.936029 31842 leader_election.cc:288] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Requested vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:18.936514 30671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fd595c79cdc43f9a8afadb6b548bf64" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:18.936751 30671 raft_consensus.cc:3037] T 4fd595c79cdc43f9a8afadb6b548bf64 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.937629 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fd595c79cdc43f9a8afadb6b548bf64" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:18.937836 30936 raft_consensus.cc:3037] T 4fd595c79cdc43f9a8afadb6b548bf64 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.939714 30671 raft_consensus.cc:2454] T 4fd595c79cdc43f9a8afadb6b548bf64 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:12:18.940254 27825 leader_election.cc:302] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a022e1a501f4d44a0468054d6c3de47, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:18.940636 31842 raft_consensus.cc:2781] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:18.940670 30936 raft_consensus.cc:2454] T 4fd595c79cdc43f9a8afadb6b548bf64 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:12:18.940865 31842 raft_consensus.cc:686] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Becoming Leader. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Running, Role: LEADER
I20240501 08:12:18.941223 31842 consensus_queue.cc:237] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:18.943240 31869 raft_consensus.cc:482] T 38c47b76fb304221b60d2b572485ad9f P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:18.943440 31869 raft_consensus.cc:504] T 38c47b76fb304221b60d2b572485ad9f P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:18.944171 31869 leader_election.cc:288] T 38c47b76fb304221b60d2b572485ad9f P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:18.944546 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c47b76fb304221b60d2b572485ad9f" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:18.944582 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c47b76fb304221b60d2b572485ad9f" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:18.944808 30537 raft_consensus.cc:2454] T 38c47b76fb304221b60d2b572485ad9f P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 0.
I20240501 08:12:18.944809 30936 raft_consensus.cc:2454] T 38c47b76fb304221b60d2b572485ad9f P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 0.
I20240501 08:12:18.945240 27958 leader_election.cc:302] T 38c47b76fb304221b60d2b572485ad9f P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:12:18.945536 31869 raft_consensus.cc:2781] T 38c47b76fb304221b60d2b572485ad9f P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:18.945629 31869 raft_consensus.cc:482] T 38c47b76fb304221b60d2b572485ad9f P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:18.945729 31869 raft_consensus.cc:3037] T 38c47b76fb304221b60d2b572485ad9f P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.947706 27620 catalog_manager.cc:5484] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af reported cstate change: term changed from 0 to 1, leader changed from <none> to f1525cbc82a24d13b65f136695c903af (127.21.84.130). New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:18.948726 31869 raft_consensus.cc:504] T 38c47b76fb304221b60d2b572485ad9f P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:18.949558 31869 leader_election.cc:288] T 38c47b76fb304221b60d2b572485ad9f P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Requested vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:18.950083 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c47b76fb304221b60d2b572485ad9f" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:18.950304 30537 raft_consensus.cc:3037] T 38c47b76fb304221b60d2b572485ad9f P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.950363 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c47b76fb304221b60d2b572485ad9f" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:18.950626 30936 raft_consensus.cc:3037] T 38c47b76fb304221b60d2b572485ad9f P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.953567 30936 raft_consensus.cc:2454] T 38c47b76fb304221b60d2b572485ad9f P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 1.
I20240501 08:12:18.954281 27957 leader_election.cc:302] T 38c47b76fb304221b60d2b572485ad9f P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a29cd590999b4f32849ff5689cd53c11, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:12:18.954619 31869 raft_consensus.cc:2781] T 38c47b76fb304221b60d2b572485ad9f P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:18.954816 31869 raft_consensus.cc:686] T 38c47b76fb304221b60d2b572485ad9f P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Becoming Leader. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Running, Role: LEADER
I20240501 08:12:18.955117 31869 consensus_queue.cc:237] T 38c47b76fb304221b60d2b572485ad9f P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:18.956233 30537 raft_consensus.cc:2454] T 38c47b76fb304221b60d2b572485ad9f P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 1.
I20240501 08:12:18.959038 27620 catalog_manager.cc:5484] T 38c47b76fb304221b60d2b572485ad9f P aad81047009a4ffcbc450b60030583b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to aad81047009a4ffcbc450b60030583b5 (127.21.84.131). New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:19.007792 27620 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:19.013532 30936 raft_consensus.cc:1261] T 0d6978a9bb3b432db852d43bd58838e6 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:19.013634 30671 raft_consensus.cc:1261] T 0d6978a9bb3b432db852d43bd58838e6 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:19.014246 31918 consensus_queue.cc:1035] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:19.014530 31966 consensus_queue.cc:1035] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:19.019305 31976 tablet.cc:1722] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:19.020020 31976 tablet.cc:1620] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:19.020081 27620 catalog_manager.cc:5576] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845) does not have the latest schema for tablet 0d6978a9bb3b432db852d43bd58838e6 (table rebalancer_dynamic_table [id=c1c4949221c344a1adaef209ccd26c0a]). Expected version 1 got 0
I20240501 08:12:19.020426 31977 tablet.cc:1722] T 0d6978a9bb3b432db852d43bd58838e6 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:19.021632 31977 tablet.cc:1620] T 0d6978a9bb3b432db852d43bd58838e6 P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:19.022675 27617 catalog_manager.cc:5576] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933) does not have the latest schema for tablet 0d6978a9bb3b432db852d43bd58838e6 (table rebalancer_dynamic_table [id=c1c4949221c344a1adaef209ccd26c0a]). Expected version 1 got 0
I20240501 08:12:19.029134 31978 tablet.cc:1722] T 0d6978a9bb3b432db852d43bd58838e6 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:19.029815 31978 tablet.cc:1620] T 0d6978a9bb3b432db852d43bd58838e6 P a29cd590999b4f32849ff5689cd53c11: MemRowSet was empty: no flush needed.
I20240501 08:12:19.032084 27613 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=c1c4949221c344a1adaef209ccd26c0a] alter complete (version 1)
I20240501 08:12:19.138527 27613 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:19.146111 31976 tablet.cc:1722] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:19.146543 31978 tablet.cc:1722] T 0d6978a9bb3b432db852d43bd58838e6 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:19.146663 31976 tablet.cc:1620] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:19.147107 31978 tablet.cc:1620] T 0d6978a9bb3b432db852d43bd58838e6 P a29cd590999b4f32849ff5689cd53c11: MemRowSet was empty: no flush needed.
I20240501 08:12:19.147512 31977 tablet.cc:1722] T 0d6978a9bb3b432db852d43bd58838e6 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:19.147984 31977 tablet.cc:1620] T 0d6978a9bb3b432db852d43bd58838e6 P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:19.155575 27620 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=c1c4949221c344a1adaef209ccd26c0a] alter complete (version 2)
I20240501 08:12:19.258193 27613 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:19.267273 31976 tablet.cc:1722] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:19.267812 31977 tablet.cc:1722] T 0d6978a9bb3b432db852d43bd58838e6 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:19.268667 31978 tablet.cc:1722] T 0d6978a9bb3b432db852d43bd58838e6 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:19.278697 27620 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table_new_name [id=c1c4949221c344a1adaef209ccd26c0a] alter complete (version 3)
I20240501 08:12:19.341548 31968 consensus_queue.cc:1035] T 38c47b76fb304221b60d2b572485ad9f P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:19.347779 31985 consensus_queue.cc:1035] T 38c47b76fb304221b60d2b572485ad9f P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:19.357228 31842 consensus_queue.cc:1035] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:19.366377 31921 consensus_queue.cc:1035] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:19.383958 27620 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:19.384208 27620 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:19.388875 27620 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 0d6978a9bb3b432db852d43bd58838e6
I20240501 08:12:19.389854 30916 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d6978a9bb3b432db852d43bd58838e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:19 UTC) from {username='slave'} at 127.0.0.1:54036
I20240501 08:12:19.389959 30517 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d6978a9bb3b432db852d43bd58838e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:19 UTC) from {username='slave'} at 127.0.0.1:45078
I20240501 08:12:19.390092 30651 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d6978a9bb3b432db852d43bd58838e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:19 UTC) from {username='slave'} at 127.0.0.1:57156
I20240501 08:12:19.392657 31991 tablet_replica.cc:330] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6: stopping tablet replica
I20240501 08:12:19.393090 31991 raft_consensus.cc:2229] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:19.393714 31991 raft_consensus.cc:2258] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:19.395073 31991 ts_tablet_manager.cc:1875] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:19.399330 31990 tablet_replica.cc:330] T 0d6978a9bb3b432db852d43bd58838e6 P a29cd590999b4f32849ff5689cd53c11: stopping tablet replica
I20240501 08:12:19.399571 31990 raft_consensus.cc:2229] T 0d6978a9bb3b432db852d43bd58838e6 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:19.401748 31992 tablet_replica.cc:330] T 0d6978a9bb3b432db852d43bd58838e6 P 9a022e1a501f4d44a0468054d6c3de47: stopping tablet replica
I20240501 08:12:19.402081 31992 raft_consensus.cc:2229] T 0d6978a9bb3b432db852d43bd58838e6 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:19.402158 31990 raft_consensus.cc:2258] T 0d6978a9bb3b432db852d43bd58838e6 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:19.402339 31992 raft_consensus.cc:2258] T 0d6978a9bb3b432db852d43bd58838e6 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:19.403870 31990 ts_tablet_manager.cc:1875] T 0d6978a9bb3b432db852d43bd58838e6 P a29cd590999b4f32849ff5689cd53c11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:19.404956 31991 ts_tablet_manager.cc:1888] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:19.405154 31991 log.cc:1196] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/wals/0d6978a9bb3b432db852d43bd58838e6
I20240501 08:12:19.405696 31991 ts_tablet_manager.cc:1909] T 0d6978a9bb3b432db852d43bd58838e6 P 31811708c6774037b82df44024b229b6: Deleting consensus metadata
I20240501 08:12:19.406885 27607 catalog_manager.cc:4830] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): tablet 0d6978a9bb3b432db852d43bd58838e6 (table rebalancer_dynamic_table_new_name [id=c1c4949221c344a1adaef209ccd26c0a]) successfully deleted
I20240501 08:12:19.407013 31992 ts_tablet_manager.cc:1875] T 0d6978a9bb3b432db852d43bd58838e6 P 9a022e1a501f4d44a0468054d6c3de47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:19.412711 31992 ts_tablet_manager.cc:1888] T 0d6978a9bb3b432db852d43bd58838e6 P 9a022e1a501f4d44a0468054d6c3de47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:19.412830 31992 log.cc:1196] T 0d6978a9bb3b432db852d43bd58838e6 P 9a022e1a501f4d44a0468054d6c3de47: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/wals/0d6978a9bb3b432db852d43bd58838e6
I20240501 08:12:19.413220 31992 ts_tablet_manager.cc:1909] T 0d6978a9bb3b432db852d43bd58838e6 P 9a022e1a501f4d44a0468054d6c3de47: Deleting consensus metadata
I20240501 08:12:19.414161 27604 catalog_manager.cc:4830] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): tablet 0d6978a9bb3b432db852d43bd58838e6 (table rebalancer_dynamic_table_new_name [id=c1c4949221c344a1adaef209ccd26c0a]) successfully deleted
I20240501 08:12:19.414341 31990 ts_tablet_manager.cc:1888] T 0d6978a9bb3b432db852d43bd58838e6 P a29cd590999b4f32849ff5689cd53c11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:19.414464 31990 log.cc:1196] T 0d6978a9bb3b432db852d43bd58838e6 P a29cd590999b4f32849ff5689cd53c11: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/wals/0d6978a9bb3b432db852d43bd58838e6
I20240501 08:12:19.414916 31990 ts_tablet_manager.cc:1909] T 0d6978a9bb3b432db852d43bd58838e6 P a29cd590999b4f32849ff5689cd53c11: Deleting consensus metadata
I20240501 08:12:19.418907 27606 catalog_manager.cc:4830] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): tablet 0d6978a9bb3b432db852d43bd58838e6 (table rebalancer_dynamic_table_new_name [id=c1c4949221c344a1adaef209ccd26c0a]) successfully deleted
I20240501 08:12:19.420436 31968 consensus_queue.cc:1035] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:19.424758 31985 consensus_queue.cc:1035] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:19.469318 27620 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_extra_table_32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:19.479563 30651 tablet_service.cc:1467] Processing CreateTablet for tablet 40a4b09831f84433b6dc126b8445cf41 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=62ce67b20f6e4a2b80c594fcbe8f131e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:19.479544 27741 tablet_service.cc:1467] Processing CreateTablet for tablet 40a4b09831f84433b6dc126b8445cf41 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=62ce67b20f6e4a2b80c594fcbe8f131e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:19.479844 30784 tablet_service.cc:1467] Processing CreateTablet for tablet 40a4b09831f84433b6dc126b8445cf41 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=62ce67b20f6e4a2b80c594fcbe8f131e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:19.480018 30651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40a4b09831f84433b6dc126b8445cf41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:19.480034 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40a4b09831f84433b6dc126b8445cf41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:19.480252 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40a4b09831f84433b6dc126b8445cf41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:19.484970 31996 tablet_bootstrap.cc:492] T 40a4b09831f84433b6dc126b8445cf41 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:19.488276 31996 tablet_bootstrap.cc:654] T 40a4b09831f84433b6dc126b8445cf41 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:19.488911 30652 tablet_service.cc:1467] Processing CreateTablet for tablet c0e30c62c2d441618195f30f15647487 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=62ce67b20f6e4a2b80c594fcbe8f131e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:19.489055 30916 tablet_service.cc:1467] Processing CreateTablet for tablet c0e30c62c2d441618195f30f15647487 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=62ce67b20f6e4a2b80c594fcbe8f131e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:19.489214 27740 tablet_service.cc:1467] Processing CreateTablet for tablet c0e30c62c2d441618195f30f15647487 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=62ce67b20f6e4a2b80c594fcbe8f131e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:19.489410 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0e30c62c2d441618195f30f15647487. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:19.489573 30916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0e30c62c2d441618195f30f15647487. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:19.489678 27740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0e30c62c2d441618195f30f15647487. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:19.490345 30917 tablet_service.cc:1467] Processing CreateTablet for tablet 82d6e5e399744b96a66373240ce1764d (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=62ce67b20f6e4a2b80c594fcbe8f131e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:19.490790 30917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82d6e5e399744b96a66373240ce1764d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:19.490789 27739 tablet_service.cc:1467] Processing CreateTablet for tablet 82d6e5e399744b96a66373240ce1764d (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=62ce67b20f6e4a2b80c594fcbe8f131e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:19.491204 27739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82d6e5e399744b96a66373240ce1764d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:19.495672 30517 tablet_service.cc:1467] Processing CreateTablet for tablet 82d6e5e399744b96a66373240ce1764d (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=62ce67b20f6e4a2b80c594fcbe8f131e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:19.496169 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82d6e5e399744b96a66373240ce1764d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:19.500404 27620 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:19.504503 31997 tablet_bootstrap.cc:492] T 40a4b09831f84433b6dc126b8445cf41 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:19.506436 31996 tablet_bootstrap.cc:492] T 40a4b09831f84433b6dc126b8445cf41 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:19.506611 31996 ts_tablet_manager.cc:1367] T 40a4b09831f84433b6dc126b8445cf41 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.007s
I20240501 08:12:19.507035 31997 tablet_bootstrap.cc:654] T 40a4b09831f84433b6dc126b8445cf41 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:19.507450 31996 raft_consensus.cc:348] T 40a4b09831f84433b6dc126b8445cf41 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:19.507681 31996 raft_consensus.cc:374] T 40a4b09831f84433b6dc126b8445cf41 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:19.507781 31996 raft_consensus.cc:729] T 40a4b09831f84433b6dc126b8445cf41 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.508069 31996 consensus_queue.cc:260] T 40a4b09831f84433b6dc126b8445cf41 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:19.512006 31996 ts_tablet_manager.cc:1398] T 40a4b09831f84433b6dc126b8445cf41 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20240501 08:12:19.514181 30782 tablet_service.cc:1467] Processing CreateTablet for tablet 2ea04e3875f04e6fadec67ddea381188 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6f2944ce48b549c1831168a409c48224]), partition=
I20240501 08:12:19.514577 30782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ea04e3875f04e6fadec67ddea381188. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:19.514631 32000 tablet_bootstrap.cc:492] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:19.515419 30916 tablet_service.cc:1467] Processing CreateTablet for tablet 2ea04e3875f04e6fadec67ddea381188 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6f2944ce48b549c1831168a409c48224]), partition=
I20240501 08:12:19.515997 30916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ea04e3875f04e6fadec67ddea381188. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:19.517668 30514 tablet_service.cc:1467] Processing CreateTablet for tablet 2ea04e3875f04e6fadec67ddea381188 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6f2944ce48b549c1831168a409c48224]), partition=
I20240501 08:12:19.518085 30514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ea04e3875f04e6fadec67ddea381188. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:19.523969 32000 tablet_bootstrap.cc:654] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:19.531392 31996 tablet_bootstrap.cc:492] T c0e30c62c2d441618195f30f15647487 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:19.531599 32002 tablet_bootstrap.cc:492] T 82d6e5e399744b96a66373240ce1764d P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:19.533068 32002 tablet_bootstrap.cc:654] T 82d6e5e399744b96a66373240ce1764d P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:19.533224 31996 tablet_bootstrap.cc:654] T c0e30c62c2d441618195f30f15647487 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:19.530812 31998 tablet_bootstrap.cc:492] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:19.539346 32002 tablet_bootstrap.cc:492] T 82d6e5e399744b96a66373240ce1764d P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:19.539501 32002 ts_tablet_manager.cc:1367] T 82d6e5e399744b96a66373240ce1764d P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20240501 08:12:19.540253 32002 raft_consensus.cc:348] T 82d6e5e399744b96a66373240ce1764d P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:19.540429 32002 raft_consensus.cc:374] T 82d6e5e399744b96a66373240ce1764d P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:19.541558 31998 tablet_bootstrap.cc:654] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:19.541921 31996 tablet_bootstrap.cc:492] T c0e30c62c2d441618195f30f15647487 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:19.542045 32002 raft_consensus.cc:729] T 82d6e5e399744b96a66373240ce1764d P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.542246 31996 ts_tablet_manager.cc:1367] T c0e30c62c2d441618195f30f15647487 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.002s
I20240501 08:12:19.542596 32002 consensus_queue.cc:260] T 82d6e5e399744b96a66373240ce1764d P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:19.543082 31996 raft_consensus.cc:348] T c0e30c62c2d441618195f30f15647487 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:19.543283 32002 ts_tablet_manager.cc:1398] T 82d6e5e399744b96a66373240ce1764d P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 08:12:19.543355 31996 raft_consensus.cc:374] T c0e30c62c2d441618195f30f15647487 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:19.543484 31996 raft_consensus.cc:729] T c0e30c62c2d441618195f30f15647487 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.543615 32002 tablet_bootstrap.cc:492] T 2ea04e3875f04e6fadec67ddea381188 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:19.543844 31996 consensus_queue.cc:260] T c0e30c62c2d441618195f30f15647487 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:19.545490 32002 tablet_bootstrap.cc:654] T 2ea04e3875f04e6fadec67ddea381188 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:19.548647 31997 tablet_bootstrap.cc:492] T 40a4b09831f84433b6dc126b8445cf41 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:12:19.549202 31997 ts_tablet_manager.cc:1367] T 40a4b09831f84433b6dc126b8445cf41 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.045s user 0.010s sys 0.011s
I20240501 08:12:19.549747 31996 ts_tablet_manager.cc:1398] T c0e30c62c2d441618195f30f15647487 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 08:12:19.550101 31997 raft_consensus.cc:348] T 40a4b09831f84433b6dc126b8445cf41 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:19.550323 31997 raft_consensus.cc:374] T 40a4b09831f84433b6dc126b8445cf41 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:19.550413 31997 raft_consensus.cc:729] T 40a4b09831f84433b6dc126b8445cf41 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.550658 31997 consensus_queue.cc:260] T 40a4b09831f84433b6dc126b8445cf41 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:19.551759 31997 ts_tablet_manager.cc:1398] T 40a4b09831f84433b6dc126b8445cf41 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:19.552044 31997 tablet_bootstrap.cc:492] T c0e30c62c2d441618195f30f15647487 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:19.553995 31997 tablet_bootstrap.cc:654] T c0e30c62c2d441618195f30f15647487 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:19.555846 32000 tablet_bootstrap.cc:492] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:19.556084 32000 ts_tablet_manager.cc:1367] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.002s
I20240501 08:12:19.556929 32000 raft_consensus.cc:348] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:19.557253 32000 raft_consensus.cc:374] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:19.557394 32000 raft_consensus.cc:729] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.557674 32000 consensus_queue.cc:260] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:19.558393 32000 ts_tablet_manager.cc:1398] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:19.560274 31997 tablet_bootstrap.cc:492] T c0e30c62c2d441618195f30f15647487 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:12:19.561070 31997 ts_tablet_manager.cc:1367] T c0e30c62c2d441618195f30f15647487 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.002s
I20240501 08:12:19.561782 31998 tablet_bootstrap.cc:492] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:19.562018 31998 ts_tablet_manager.cc:1367] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.000s
I20240501 08:12:19.562095 31997 raft_consensus.cc:348] T c0e30c62c2d441618195f30f15647487 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:19.562389 31997 raft_consensus.cc:374] T c0e30c62c2d441618195f30f15647487 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:19.562491 31997 raft_consensus.cc:729] T c0e30c62c2d441618195f30f15647487 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.563170 31998 raft_consensus.cc:348] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:19.563445 31998 raft_consensus.cc:374] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:19.563593 31998 raft_consensus.cc:729] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.562796 31997 consensus_queue.cc:260] T c0e30c62c2d441618195f30f15647487 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:19.564868 31997 ts_tablet_manager.cc:1398] T c0e30c62c2d441618195f30f15647487 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:19.566037 31997 tablet_bootstrap.cc:492] T 82d6e5e399744b96a66373240ce1764d P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:19.566999 32000 tablet_bootstrap.cc:492] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:19.563905 31998 consensus_queue.cc:260] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:19.567698 31997 tablet_bootstrap.cc:654] T 82d6e5e399744b96a66373240ce1764d P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:19.568218 31998 ts_tablet_manager.cc:1398] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:12:19.568586 31998 tablet_bootstrap.cc:492] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:19.571741 31998 tablet_bootstrap.cc:654] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:19.572984 32002 tablet_bootstrap.cc:492] T 2ea04e3875f04e6fadec67ddea381188 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:19.573196 32002 ts_tablet_manager.cc:1367] T 2ea04e3875f04e6fadec67ddea381188 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.012s
I20240501 08:12:19.575052 32002 raft_consensus.cc:348] T 2ea04e3875f04e6fadec67ddea381188 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:19.575721 32000 tablet_bootstrap.cc:654] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:19.575471 32002 raft_consensus.cc:374] T 2ea04e3875f04e6fadec67ddea381188 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:19.576359 32002 raft_consensus.cc:729] T 2ea04e3875f04e6fadec67ddea381188 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.576638 32002 consensus_queue.cc:260] T 2ea04e3875f04e6fadec67ddea381188 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:19.577605 32002 ts_tablet_manager.cc:1398] T 2ea04e3875f04e6fadec67ddea381188 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:12:19.578351 31997 tablet_bootstrap.cc:492] T 82d6e5e399744b96a66373240ce1764d P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:12:19.580868 31997 ts_tablet_manager.cc:1367] T 82d6e5e399744b96a66373240ce1764d P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240501 08:12:19.581689 31997 raft_consensus.cc:348] T 82d6e5e399744b96a66373240ce1764d P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:19.581991 31997 raft_consensus.cc:374] T 82d6e5e399744b96a66373240ce1764d P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:19.582130 31997 raft_consensus.cc:729] T 82d6e5e399744b96a66373240ce1764d P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.582408 31997 consensus_queue.cc:260] T 82d6e5e399744b96a66373240ce1764d P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:19.584045 31997 ts_tablet_manager.cc:1398] T 82d6e5e399744b96a66373240ce1764d P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 08:12:19.585498 31998 tablet_bootstrap.cc:492] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:19.585667 31998 ts_tablet_manager.cc:1367] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20240501 08:12:19.586387 31998 raft_consensus.cc:348] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:19.586565 31998 raft_consensus.cc:374] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:19.586647 31998 raft_consensus.cc:729] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.586994 31998 consensus_queue.cc:260] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:19.587545 31998 ts_tablet_manager.cc:1398] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:19.591065 32000 tablet_bootstrap.cc:492] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:19.591261 32000 ts_tablet_manager.cc:1367] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.001s
I20240501 08:12:19.592864 32000 raft_consensus.cc:348] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:19.593120 32000 raft_consensus.cc:374] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:19.593199 32000 raft_consensus.cc:729] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.593423 32000 consensus_queue.cc:260] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:19.594816 32000 ts_tablet_manager.cc:1398] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 08:12:19.595144 32000 tablet_bootstrap.cc:492] T 2ea04e3875f04e6fadec67ddea381188 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:19.596510 32000 tablet_bootstrap.cc:654] T 2ea04e3875f04e6fadec67ddea381188 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:19.603365 32000 tablet_bootstrap.cc:492] T 2ea04e3875f04e6fadec67ddea381188 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:19.603531 32000 ts_tablet_manager.cc:1367] T 2ea04e3875f04e6fadec67ddea381188 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.001s
I20240501 08:12:19.604290 32000 raft_consensus.cc:348] T 2ea04e3875f04e6fadec67ddea381188 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:19.604507 32000 raft_consensus.cc:374] T 2ea04e3875f04e6fadec67ddea381188 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:19.604632 32000 raft_consensus.cc:729] T 2ea04e3875f04e6fadec67ddea381188 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.604884 32000 consensus_queue.cc:260] T 2ea04e3875f04e6fadec67ddea381188 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:19.605556 32000 ts_tablet_manager.cc:1398] T 2ea04e3875f04e6fadec67ddea381188 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 08:12:19.655731 31641 raft_consensus.cc:482] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:19.655913 31641 raft_consensus.cc:504] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:19.656718 31641 leader_election.cc:288] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:19.657132 30671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e30c62c2d441618195f30f15647487" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:19.657364 30671 raft_consensus.cc:2454] T c0e30c62c2d441618195f30f15647487 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:19.657418 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e30c62c2d441618195f30f15647487" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:12:19.657644 27759 raft_consensus.cc:2454] T c0e30c62c2d441618195f30f15647487 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:19.658115 30870 leader_election.cc:302] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a022e1a501f4d44a0468054d6c3de47, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:19.658413 31641 raft_consensus.cc:2781] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:19.659025 31641 raft_consensus.cc:482] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:19.659138 31641 raft_consensus.cc:3037] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:19.661733 31641 raft_consensus.cc:504] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:19.662559 31641 leader_election.cc:288] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Requested vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:19.663138 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e30c62c2d441618195f30f15647487" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:12:19.663373 27759 raft_consensus.cc:3037] T c0e30c62c2d441618195f30f15647487 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:19.663296 30671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e30c62c2d441618195f30f15647487" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:19.663528 30671 raft_consensus.cc:3037] T c0e30c62c2d441618195f30f15647487 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:19.666188 27759 raft_consensus.cc:2454] T c0e30c62c2d441618195f30f15647487 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:19.666220 30671 raft_consensus.cc:2454] T c0e30c62c2d441618195f30f15647487 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:19.666599 30872 leader_election.cc:302] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:19.666934 31641 raft_consensus.cc:2781] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:19.667320 31641 raft_consensus.cc:686] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Becoming Leader. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:12:19.667587 31641 consensus_queue.cc:237] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:19.671070 27617 catalog_manager.cc:5484] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: term changed from 0 to 1, leader changed from <none> to a29cd590999b4f32849ff5689cd53c11 (127.21.84.136). New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:19.677546 31743 raft_consensus.cc:482] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:19.677721 31743 raft_consensus.cc:504] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:19.678470 31743 leader_election.cc:288] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:19.679049 30671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40a4b09831f84433b6dc126b8445cf41" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:19.679260 30671 raft_consensus.cc:2454] T 40a4b09831f84433b6dc126b8445cf41 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:19.679689 30737 leader_election.cc:302] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:19.679811 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40a4b09831f84433b6dc126b8445cf41" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:12:19.679960 31743 raft_consensus.cc:2781] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:19.680042 27759 raft_consensus.cc:2454] T 40a4b09831f84433b6dc126b8445cf41 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:19.680073 31743 raft_consensus.cc:482] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:19.680162 31743 raft_consensus.cc:3037] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:19.680490 31641 raft_consensus.cc:482] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:19.680717 31641 raft_consensus.cc:504] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:19.681496 31641 leader_election.cc:288] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:19.682042 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82d6e5e399744b96a66373240ce1764d" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:19.682250 30536 raft_consensus.cc:2454] T 82d6e5e399744b96a66373240ce1764d P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:19.682685 30873 leader_election.cc:302] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:19.682751 31743 raft_consensus.cc:504] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:19.682946 31641 raft_consensus.cc:2781] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:19.683055 31641 raft_consensus.cc:482] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:19.683354 31641 raft_consensus.cc:3037] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:19.683452 31743 leader_election.cc:288] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Requested vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:19.684551 30671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40a4b09831f84433b6dc126b8445cf41" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:19.684602 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82d6e5e399744b96a66373240ce1764d" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:12:19.684861 27759 raft_consensus.cc:2454] T 82d6e5e399744b96a66373240ce1764d P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:19.684885 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40a4b09831f84433b6dc126b8445cf41" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:12:19.685158 27761 raft_consensus.cc:3037] T 40a4b09831f84433b6dc126b8445cf41 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:19.686067 30671 raft_consensus.cc:3037] T 40a4b09831f84433b6dc126b8445cf41 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:19.687353 27761 raft_consensus.cc:2454] T 40a4b09831f84433b6dc126b8445cf41 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:19.687693 30739 leader_election.cc:302] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 64998323a1f4482d8704bf20594d13a5; no voters:
I20240501 08:12:19.687513 31641 raft_consensus.cc:504] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:19.687958 31743 raft_consensus.cc:2781] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:19.688223 31743 raft_consensus.cc:686] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Becoming Leader. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Running, Role: LEADER
I20240501 08:12:19.688593 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82d6e5e399744b96a66373240ce1764d" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:19.688794 31641 leader_election.cc:288] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Requested vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:19.688782 30671 raft_consensus.cc:2454] T 40a4b09831f84433b6dc126b8445cf41 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:19.688823 30536 raft_consensus.cc:3037] T 82d6e5e399744b96a66373240ce1764d P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:19.688571 31743 consensus_queue.cc:237] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:19.691614 30536 raft_consensus.cc:2454] T 82d6e5e399744b96a66373240ce1764d P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:19.692309 30873 leader_election.cc:302] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:19.692567 31641 raft_consensus.cc:2781] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:19.692732 31641 raft_consensus.cc:686] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Becoming Leader. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:12:19.693020 31641 consensus_queue.cc:237] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:19.693650 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82d6e5e399744b96a66373240ce1764d" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:12:19.693883 27761 raft_consensus.cc:3037] T 82d6e5e399744b96a66373240ce1764d P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:19.696705 27761 raft_consensus.cc:2454] T 82d6e5e399744b96a66373240ce1764d P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:19.706511 27617 catalog_manager.cc:5484] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 reported cstate change: term changed from 0 to 1, leader changed from <none> to a29cd590999b4f32849ff5689cd53c11 (127.21.84.136). New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:19.709686 27620 catalog_manager.cc:5484] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64998323a1f4482d8704bf20594d13a5 (127.21.84.135). New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:19.778005 31743 raft_consensus.cc:482] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:19.778239 31743 raft_consensus.cc:504] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:19.778908 31743 leader_election.cc:288] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:19.779342 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea04e3875f04e6fadec67ddea381188" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:19.779596 30536 raft_consensus.cc:2454] T 2ea04e3875f04e6fadec67ddea381188 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:19.779721 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea04e3875f04e6fadec67ddea381188" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:19.779969 30936 raft_consensus.cc:2454] T 2ea04e3875f04e6fadec67ddea381188 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:19.780053 30740 leader_election.cc:302] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 64998323a1f4482d8704bf20594d13a5; no voters:
I20240501 08:12:19.780349 31743 raft_consensus.cc:2781] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:19.780450 31743 raft_consensus.cc:482] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:19.780561 31743 raft_consensus.cc:3037] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:19.782521 31743 raft_consensus.cc:504] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:19.783167 31743 leader_election.cc:288] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Requested vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:19.783586 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea04e3875f04e6fadec67ddea381188" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:19.783579 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea04e3875f04e6fadec67ddea381188" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:19.783779 30536 raft_consensus.cc:3037] T 2ea04e3875f04e6fadec67ddea381188 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:19.783804 30936 raft_consensus.cc:3037] T 2ea04e3875f04e6fadec67ddea381188 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:19.785813 30536 raft_consensus.cc:2454] T 2ea04e3875f04e6fadec67ddea381188 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:19.785907 30936 raft_consensus.cc:2454] T 2ea04e3875f04e6fadec67ddea381188 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:19.786278 30740 leader_election.cc:302] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 64998323a1f4482d8704bf20594d13a5; no voters:
I20240501 08:12:19.786548 31743 raft_consensus.cc:2781] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:19.786702 31743 raft_consensus.cc:686] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Becoming Leader. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Running, Role: LEADER
I20240501 08:12:19.786947 31743 consensus_queue.cc:237] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:19.790506 27620 catalog_manager.cc:5484] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64998323a1f4482d8704bf20594d13a5 (127.21.84.135). New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:19.920601 27620 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:19.928181 30936 raft_consensus.cc:1261] T 2ea04e3875f04e6fadec67ddea381188 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:19.928759 31743 consensus_queue.cc:1035] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:19.929651 30537 raft_consensus.cc:1261] T 2ea04e3875f04e6fadec67ddea381188 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:19.930599 31743 consensus_queue.cc:1035] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:19.947510 27620 catalog_manager.cc:5576] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293) does not have the latest schema for tablet 2ea04e3875f04e6fadec67ddea381188 (table rebalancer_dynamic_table [id=6f2944ce48b549c1831168a409c48224]). Expected version 1 got 0
I20240501 08:12:19.949131 32025 tablet.cc:1722] T 2ea04e3875f04e6fadec67ddea381188 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:19.949921 32025 tablet.cc:1620] T 2ea04e3875f04e6fadec67ddea381188 P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:19.952827 27620 catalog_manager.cc:5576] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293) does not have the latest schema for tablet 2ea04e3875f04e6fadec67ddea381188 (table rebalancer_dynamic_table [id=6f2944ce48b549c1831168a409c48224]). Expected version 1 got 0
I20240501 08:12:19.961781 27617 catalog_manager.cc:5576] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845) does not have the latest schema for tablet 2ea04e3875f04e6fadec67ddea381188 (table rebalancer_dynamic_table [id=6f2944ce48b549c1831168a409c48224]). Expected version 1 got 0
I20240501 08:12:19.962407 32024 tablet.cc:1722] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:19.963022 32024 tablet.cc:1620] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5: MemRowSet was empty: no flush needed.
I20240501 08:12:19.964172 27620 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=6f2944ce48b549c1831168a409c48224] alter complete (version 1)
I20240501 08:12:19.965591 32027 tablet.cc:1722] T 2ea04e3875f04e6fadec67ddea381188 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:19.966228 32027 tablet.cc:1620] T 2ea04e3875f04e6fadec67ddea381188 P a29cd590999b4f32849ff5689cd53c11: MemRowSet was empty: no flush needed.
I20240501 08:12:19.968272 27620 catalog_manager.cc:5576] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845) does not have the latest schema for tablet 2ea04e3875f04e6fadec67ddea381188 (table rebalancer_dynamic_table [id=6f2944ce48b549c1831168a409c48224]). Expected version 1 got 0
I20240501 08:12:20.066434 27620 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:20.074270 32024 tablet.cc:1722] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:20.074906 32024 tablet.cc:1620] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5: MemRowSet was empty: no flush needed.
I20240501 08:12:20.075745 32025 tablet.cc:1722] T 2ea04e3875f04e6fadec67ddea381188 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:20.076256 32025 tablet.cc:1620] T 2ea04e3875f04e6fadec67ddea381188 P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:20.076573 32027 tablet.cc:1722] T 2ea04e3875f04e6fadec67ddea381188 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:20.077119 32027 tablet.cc:1620] T 2ea04e3875f04e6fadec67ddea381188 P a29cd590999b4f32849ff5689cd53c11: MemRowSet was empty: no flush needed.
I20240501 08:12:20.085038 27617 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=6f2944ce48b549c1831168a409c48224] alter complete (version 2)
I20240501 08:12:20.099090 32013 consensus_queue.cc:1035] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:20.112401 31641 consensus_queue.cc:1035] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:20.164722 32028 consensus_queue.cc:1035] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:20.168975 31748 consensus_queue.cc:1035] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:20.187290 31641 consensus_queue.cc:1035] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:20.189728 27613 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:20.193149 32013 consensus_queue.cc:1035] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:20.201210 32024 tablet.cc:1722] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:20.202041 32025 tablet.cc:1722] T 2ea04e3875f04e6fadec67ddea381188 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:20.206983 32027 tablet.cc:1722] T 2ea04e3875f04e6fadec67ddea381188 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:20.210700 27611 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table_new_name [id=6f2944ce48b549c1831168a409c48224] alter complete (version 3)
I20240501 08:12:20.239792 27613 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_extra_table_34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:20.250535 30652 tablet_service.cc:1467] Processing CreateTablet for tablet 95cb5060152e4237988a4c5723a62601 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=1d72487b1d1a4310bbfd3ad3afddbee9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:20.250564 30916 tablet_service.cc:1467] Processing CreateTablet for tablet 95cb5060152e4237988a4c5723a62601 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=1d72487b1d1a4310bbfd3ad3afddbee9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:20.251045 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95cb5060152e4237988a4c5723a62601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.251049 30916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95cb5060152e4237988a4c5723a62601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.254390 28002 tablet_service.cc:1467] Processing CreateTablet for tablet 95cb5060152e4237988a4c5723a62601 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=1d72487b1d1a4310bbfd3ad3afddbee9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:20.254619 30514 tablet_service.cc:1467] Processing CreateTablet for tablet 05eac5f1b08945f8b7b8c78d0f4d3d3a (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=1d72487b1d1a4310bbfd3ad3afddbee9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:20.254778 28002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95cb5060152e4237988a4c5723a62601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.255519 30514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05eac5f1b08945f8b7b8c78d0f4d3d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.258898 27872 tablet_service.cc:1467] Processing CreateTablet for tablet 05eac5f1b08945f8b7b8c78d0f4d3d3a (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=1d72487b1d1a4310bbfd3ad3afddbee9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:20.259121 30917 tablet_service.cc:1467] Processing CreateTablet for tablet 00a73d23b12d48a2966f94789db4369a (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=1d72487b1d1a4310bbfd3ad3afddbee9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:20.259341 27872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05eac5f1b08945f8b7b8c78d0f4d3d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.259265 30651 tablet_service.cc:1467] Processing CreateTablet for tablet 00a73d23b12d48a2966f94789db4369a (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=1d72487b1d1a4310bbfd3ad3afddbee9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:20.259577 30917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00a73d23b12d48a2966f94789db4369a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.259603 30651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00a73d23b12d48a2966f94789db4369a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.262347 30915 tablet_service.cc:1467] Processing CreateTablet for tablet 05eac5f1b08945f8b7b8c78d0f4d3d3a (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=1d72487b1d1a4310bbfd3ad3afddbee9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:20.262764 30915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05eac5f1b08945f8b7b8c78d0f4d3d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.263558 30784 tablet_service.cc:1467] Processing CreateTablet for tablet 00a73d23b12d48a2966f94789db4369a (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=1d72487b1d1a4310bbfd3ad3afddbee9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:20.263979 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00a73d23b12d48a2966f94789db4369a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.279875 32040 tablet_bootstrap.cc:492] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:20.281268 32038 tablet_bootstrap.cc:492] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:20.282559 32040 tablet_bootstrap.cc:654] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:20.283463 32038 tablet_bootstrap.cc:654] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:20.291435 32040 tablet_bootstrap.cc:492] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:20.291720 32040 ts_tablet_manager.cc:1367] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20240501 08:12:20.291926 32038 tablet_bootstrap.cc:492] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:12:20.292171 32038 ts_tablet_manager.cc:1367] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20240501 08:12:20.292595 32040 raft_consensus.cc:348] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:20.292807 32040 raft_consensus.cc:374] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:20.292905 32040 raft_consensus.cc:729] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.293120 32040 consensus_queue.cc:260] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:20.293810 32040 ts_tablet_manager.cc:1398] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:20.295068 32038 raft_consensus.cc:348] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:20.295418 32038 raft_consensus.cc:374] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:20.295557 32038 raft_consensus.cc:729] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.295904 32042 tablet_bootstrap.cc:492] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:20.295851 32038 consensus_queue.cc:260] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:20.296739 32038 ts_tablet_manager.cc:1398] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 08:12:20.297344 32039 tablet_bootstrap.cc:492] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:20.298072 32041 tablet_bootstrap.cc:492] T 95cb5060152e4237988a4c5723a62601 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:20.298223 32045 tablet_bootstrap.cc:492] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:20.299158 32039 tablet_bootstrap.cc:654] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:20.301220 32042 tablet_bootstrap.cc:654] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:20.308115 32045 tablet_bootstrap.cc:654] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:20.309581 32041 tablet_bootstrap.cc:654] T 95cb5060152e4237988a4c5723a62601 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:20.312023 32045 tablet_bootstrap.cc:492] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:20.312007 32042 tablet_bootstrap.cc:492] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:12:20.312312 32045 ts_tablet_manager.cc:1367] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240501 08:12:20.312312 32042 ts_tablet_manager.cc:1367] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240501 08:12:20.313205 32045 raft_consensus.cc:348] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:20.313513 32045 raft_consensus.cc:374] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:20.313668 32045 raft_consensus.cc:729] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.313699 32041 tablet_bootstrap.cc:492] T 95cb5060152e4237988a4c5723a62601 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:20.313906 32041 ts_tablet_manager.cc:1367] T 95cb5060152e4237988a4c5723a62601 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.003s
I20240501 08:12:20.314100 32045 consensus_queue.cc:260] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:20.314929 32041 raft_consensus.cc:348] T 95cb5060152e4237988a4c5723a62601 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:20.315207 32041 raft_consensus.cc:374] T 95cb5060152e4237988a4c5723a62601 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:20.315153 27620 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:20.315322 32041 raft_consensus.cc:729] T 95cb5060152e4237988a4c5723a62601 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.315387 27620 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:20.315604 32041 consensus_queue.cc:260] T 95cb5060152e4237988a4c5723a62601 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:20.316334 32041 ts_tablet_manager.cc:1398] T 95cb5060152e4237988a4c5723a62601 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:20.316673 32041 tablet_bootstrap.cc:492] T 00a73d23b12d48a2966f94789db4369a P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:20.318287 32041 tablet_bootstrap.cc:654] T 00a73d23b12d48a2966f94789db4369a P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:20.318559 32045 ts_tablet_manager.cc:1398] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:12:20.320976 32039 tablet_bootstrap.cc:492] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:20.321051 27620 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 2ea04e3875f04e6fadec67ddea381188
I20240501 08:12:20.321180 32039 ts_tablet_manager.cc:1367] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.007s
I20240501 08:12:20.322234 30514 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ea04e3875f04e6fadec67ddea381188 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:20 UTC) from {username='slave'} at 127.0.0.1:45078
I20240501 08:12:20.322233 32039 raft_consensus.cc:348] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:20.322454 32039 raft_consensus.cc:374] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:20.322551 32039 raft_consensus.cc:729] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.322954 30915 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ea04e3875f04e6fadec67ddea381188 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:20 UTC) from {username='slave'} at 127.0.0.1:54036
I20240501 08:12:20.322949 30784 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ea04e3875f04e6fadec67ddea381188 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:20 UTC) from {username='slave'} at 127.0.0.1:38952
I20240501 08:12:20.322832 32039 consensus_queue.cc:260] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:20.323524 32039 ts_tablet_manager.cc:1398] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:20.327669 32041 tablet_bootstrap.cc:492] T 00a73d23b12d48a2966f94789db4369a P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:20.328013 32041 ts_tablet_manager.cc:1367] T 00a73d23b12d48a2966f94789db4369a P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240501 08:12:20.329106 32041 raft_consensus.cc:348] T 00a73d23b12d48a2966f94789db4369a P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:20.329531 32041 raft_consensus.cc:374] T 00a73d23b12d48a2966f94789db4369a P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:20.329679 32041 raft_consensus.cc:729] T 00a73d23b12d48a2966f94789db4369a P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.330121 32041 consensus_queue.cc:260] T 00a73d23b12d48a2966f94789db4369a P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:20.331128 32042 raft_consensus.cc:348] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:20.331893 32042 raft_consensus.cc:374] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:20.333920 32041 ts_tablet_manager.cc:1398] T 00a73d23b12d48a2966f94789db4369a P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240501 08:12:20.334488 32039 tablet_bootstrap.cc:492] T 00a73d23b12d48a2966f94789db4369a P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:20.336639 32039 tablet_bootstrap.cc:654] T 00a73d23b12d48a2966f94789db4369a P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:20.341367 32042 raft_consensus.cc:729] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.342213 32042 consensus_queue.cc:260] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:20.343058 32052 tablet_replica.cc:330] T 2ea04e3875f04e6fadec67ddea381188 P 31811708c6774037b82df44024b229b6: stopping tablet replica
I20240501 08:12:20.343628 32042 ts_tablet_manager.cc:1398] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.031s user 0.003s sys 0.000s
I20240501 08:12:20.346228 32054 tablet_replica.cc:330] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5: stopping tablet replica
I20240501 08:12:20.346894 32053 tablet_replica.cc:330] T 2ea04e3875f04e6fadec67ddea381188 P a29cd590999b4f32849ff5689cd53c11: stopping tablet replica
I20240501 08:12:20.347208 32053 raft_consensus.cc:2229] T 2ea04e3875f04e6fadec67ddea381188 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:20.347486 32053 raft_consensus.cc:2258] T 2ea04e3875f04e6fadec67ddea381188 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:20.347973 32054 raft_consensus.cc:2229] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:20.348485 32054 raft_consensus.cc:2258] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:20.348690 32053 ts_tablet_manager.cc:1875] T 2ea04e3875f04e6fadec67ddea381188 P a29cd590999b4f32849ff5689cd53c11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:20.348884 32052 raft_consensus.cc:2229] T 2ea04e3875f04e6fadec67ddea381188 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:20.349205 32052 raft_consensus.cc:2258] T 2ea04e3875f04e6fadec67ddea381188 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:20.352742 32054 ts_tablet_manager.cc:1875] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:20.357995 32052 ts_tablet_manager.cc:1875] T 2ea04e3875f04e6fadec67ddea381188 P 31811708c6774037b82df44024b229b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:20.360551 32054 ts_tablet_manager.cc:1888] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:20.360785 32054 log.cc:1196] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal/wals/2ea04e3875f04e6fadec67ddea381188
I20240501 08:12:20.361457 32054 ts_tablet_manager.cc:1909] T 2ea04e3875f04e6fadec67ddea381188 P 64998323a1f4482d8704bf20594d13a5: Deleting consensus metadata
I20240501 08:12:20.362496 32039 tablet_bootstrap.cc:492] T 00a73d23b12d48a2966f94789db4369a P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:20.362689 32039 ts_tablet_manager.cc:1367] T 00a73d23b12d48a2966f94789db4369a P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.018s
I20240501 08:12:20.363505 32039 raft_consensus.cc:348] T 00a73d23b12d48a2966f94789db4369a P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:20.363752 32039 raft_consensus.cc:374] T 00a73d23b12d48a2966f94789db4369a P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:20.363862 32039 raft_consensus.cc:729] T 00a73d23b12d48a2966f94789db4369a P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.364138 32039 consensus_queue.cc:260] T 00a73d23b12d48a2966f94789db4369a P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:20.364719 32039 ts_tablet_manager.cc:1398] T 00a73d23b12d48a2966f94789db4369a P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:20.365211 32039 tablet_bootstrap.cc:492] T 95cb5060152e4237988a4c5723a62601 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:20.366672 27605 catalog_manager.cc:4830] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): tablet 2ea04e3875f04e6fadec67ddea381188 (table rebalancer_dynamic_table_new_name [id=6f2944ce48b549c1831168a409c48224]) successfully deleted
I20240501 08:12:20.366837 32052 ts_tablet_manager.cc:1888] T 2ea04e3875f04e6fadec67ddea381188 P 31811708c6774037b82df44024b229b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:20.366964 32052 log.cc:1196] T 2ea04e3875f04e6fadec67ddea381188 P 31811708c6774037b82df44024b229b6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/wals/2ea04e3875f04e6fadec67ddea381188
I20240501 08:12:20.367316 32039 tablet_bootstrap.cc:654] T 95cb5060152e4237988a4c5723a62601 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:20.367425 32052 ts_tablet_manager.cc:1909] T 2ea04e3875f04e6fadec67ddea381188 P 31811708c6774037b82df44024b229b6: Deleting consensus metadata
I20240501 08:12:20.368568 27607 catalog_manager.cc:4830] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): tablet 2ea04e3875f04e6fadec67ddea381188 (table rebalancer_dynamic_table_new_name [id=6f2944ce48b549c1831168a409c48224]) successfully deleted
I20240501 08:12:20.371435 32053 ts_tablet_manager.cc:1888] T 2ea04e3875f04e6fadec67ddea381188 P a29cd590999b4f32849ff5689cd53c11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:20.371445 32039 tablet_bootstrap.cc:492] T 95cb5060152e4237988a4c5723a62601 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:20.371630 32053 log.cc:1196] T 2ea04e3875f04e6fadec67ddea381188 P a29cd590999b4f32849ff5689cd53c11: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/wals/2ea04e3875f04e6fadec67ddea381188
I20240501 08:12:20.371809 32039 ts_tablet_manager.cc:1367] T 95cb5060152e4237988a4c5723a62601 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.002s
I20240501 08:12:20.372181 32053 ts_tablet_manager.cc:1909] T 2ea04e3875f04e6fadec67ddea381188 P a29cd590999b4f32849ff5689cd53c11: Deleting consensus metadata
I20240501 08:12:20.372709 32039 raft_consensus.cc:348] T 95cb5060152e4237988a4c5723a62601 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:20.372934 32039 raft_consensus.cc:374] T 95cb5060152e4237988a4c5723a62601 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:20.373085 32039 raft_consensus.cc:729] T 95cb5060152e4237988a4c5723a62601 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.373257 27606 catalog_manager.cc:4830] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): tablet 2ea04e3875f04e6fadec67ddea381188 (table rebalancer_dynamic_table_new_name [id=6f2944ce48b549c1831168a409c48224]) successfully deleted
I20240501 08:12:20.373311 32039 consensus_queue.cc:260] T 95cb5060152e4237988a4c5723a62601 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:20.373875 32039 ts_tablet_manager.cc:1398] T 95cb5060152e4237988a4c5723a62601 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:12:20.424132 27620 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:20.431326 30784 tablet_service.cc:1467] Processing CreateTablet for tablet df8da14ba7c146a9b65d64625ccaf59a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6c3dec485395467aa63b258afc95201e]), partition=
I20240501 08:12:20.431620 28002 tablet_service.cc:1467] Processing CreateTablet for tablet df8da14ba7c146a9b65d64625ccaf59a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6c3dec485395467aa63b258afc95201e]), partition=
I20240501 08:12:20.431602 30652 tablet_service.cc:1467] Processing CreateTablet for tablet df8da14ba7c146a9b65d64625ccaf59a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6c3dec485395467aa63b258afc95201e]), partition=
I20240501 08:12:20.431735 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df8da14ba7c146a9b65d64625ccaf59a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.431950 28002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df8da14ba7c146a9b65d64625ccaf59a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.431969 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df8da14ba7c146a9b65d64625ccaf59a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.437026 32045 tablet_bootstrap.cc:492] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:20.438469 32041 tablet_bootstrap.cc:492] T df8da14ba7c146a9b65d64625ccaf59a P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:20.438513 32042 tablet_bootstrap.cc:492] T df8da14ba7c146a9b65d64625ccaf59a P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:20.438623 32045 tablet_bootstrap.cc:654] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:20.439877 32041 tablet_bootstrap.cc:654] T df8da14ba7c146a9b65d64625ccaf59a P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:20.440125 32042 tablet_bootstrap.cc:654] T df8da14ba7c146a9b65d64625ccaf59a P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:20.440724 32045 tablet_bootstrap.cc:492] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:20.440884 32045 ts_tablet_manager.cc:1367] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:12:20.441604 32041 tablet_bootstrap.cc:492] T df8da14ba7c146a9b65d64625ccaf59a P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:20.441736 32041 ts_tablet_manager.cc:1367] T df8da14ba7c146a9b65d64625ccaf59a P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240501 08:12:20.442497 32041 raft_consensus.cc:348] T df8da14ba7c146a9b65d64625ccaf59a P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:20.442718 32041 raft_consensus.cc:374] T df8da14ba7c146a9b65d64625ccaf59a P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:20.442804 32041 raft_consensus.cc:729] T df8da14ba7c146a9b65d64625ccaf59a P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.442379 32045 raft_consensus.cc:348] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:20.442942 32045 raft_consensus.cc:374] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:20.443039 32045 raft_consensus.cc:729] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.443413 32045 consensus_queue.cc:260] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:20.443717 32041 consensus_queue.cc:260] T df8da14ba7c146a9b65d64625ccaf59a P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:20.443892 32045 ts_tablet_manager.cc:1398] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240501 08:12:20.444445 32041 ts_tablet_manager.cc:1398] T df8da14ba7c146a9b65d64625ccaf59a P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:12:20.454860 31921 raft_consensus.cc:482] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:20.455052 31921 raft_consensus.cc:504] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:20.455752 31921 leader_election.cc:288] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:20.456413 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05eac5f1b08945f8b7b8c78d0f4d3d3a" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:20.456653 30537 raft_consensus.cc:2454] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:12:20.457098 27828 leader_election.cc:302] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:20.457372 31921 raft_consensus.cc:2781] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:20.457469 31921 raft_consensus.cc:482] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:20.457567 31921 raft_consensus.cc:3037] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:20.458109 32042 tablet_bootstrap.cc:492] T df8da14ba7c146a9b65d64625ccaf59a P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:12:20.458324 32042 ts_tablet_manager.cc:1367] T df8da14ba7c146a9b65d64625ccaf59a P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.008s
I20240501 08:12:20.459098 32042 raft_consensus.cc:348] T df8da14ba7c146a9b65d64625ccaf59a P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:20.459283 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05eac5f1b08945f8b7b8c78d0f4d3d3a" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:20.459401 32042 raft_consensus.cc:374] T df8da14ba7c146a9b65d64625ccaf59a P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:20.459515 32042 raft_consensus.cc:729] T df8da14ba7c146a9b65d64625ccaf59a P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.459558 30936 raft_consensus.cc:2454] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:12:20.459790 32042 consensus_queue.cc:260] T df8da14ba7c146a9b65d64625ccaf59a P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:20.460601 32042 ts_tablet_manager.cc:1398] T df8da14ba7c146a9b65d64625ccaf59a P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:20.460891 31921 raft_consensus.cc:504] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:20.461652 31921 leader_election.cc:288] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Requested vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:20.461920 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05eac5f1b08945f8b7b8c78d0f4d3d3a" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:20.462175 30537 raft_consensus.cc:3037] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:20.464512 30537 raft_consensus.cc:2454] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:12:20.465118 27828 leader_election.cc:302] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:20.465507 31921 raft_consensus.cc:2781] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:20.465658 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05eac5f1b08945f8b7b8c78d0f4d3d3a" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:20.465866 31921 raft_consensus.cc:686] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Becoming Leader. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Running, Role: LEADER
I20240501 08:12:20.465891 30936 raft_consensus.cc:3037] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:20.466230 31921 consensus_queue.cc:237] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:20.470458 30936 raft_consensus.cc:2454] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:12:20.470834 27613 catalog_manager.cc:5484] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P f1525cbc82a24d13b65f136695c903af reported cstate change: term changed from 0 to 1, leader changed from <none> to f1525cbc82a24d13b65f136695c903af (127.21.84.130). New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:20.499362 32028 raft_consensus.cc:482] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:20.499542 32028 raft_consensus.cc:504] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:20.500506 32028 leader_election.cc:288] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:20.500984 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a73d23b12d48a2966f94789db4369a" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:20.501211 30936 raft_consensus.cc:2454] T 00a73d23b12d48a2966f94789db4369a P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:20.501144 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a73d23b12d48a2966f94789db4369a" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:20.501394 30672 raft_consensus.cc:2454] T 00a73d23b12d48a2966f94789db4369a P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:20.501585 30739 leader_election.cc:302] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:20.501907 32028 raft_consensus.cc:2781] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:20.502051 32028 raft_consensus.cc:482] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:20.502156 32028 raft_consensus.cc:3037] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:20.504491 32028 raft_consensus.cc:504] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:20.505127 32028 leader_election.cc:288] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Requested vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:20.505529 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a73d23b12d48a2966f94789db4369a" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:20.505615 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a73d23b12d48a2966f94789db4369a" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:20.505710 30936 raft_consensus.cc:3037] T 00a73d23b12d48a2966f94789db4369a P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:20.505831 30672 raft_consensus.cc:3037] T 00a73d23b12d48a2966f94789db4369a P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:20.507905 30672 raft_consensus.cc:2454] T 00a73d23b12d48a2966f94789db4369a P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:20.507938 30936 raft_consensus.cc:2454] T 00a73d23b12d48a2966f94789db4369a P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:20.508311 30737 leader_election.cc:302] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:20.508607 32028 raft_consensus.cc:2781] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:20.508749 32028 raft_consensus.cc:686] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Becoming Leader. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Running, Role: LEADER
I20240501 08:12:20.509050 32028 consensus_queue.cc:237] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:20.512246 27613 catalog_manager.cc:5484] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64998323a1f4482d8704bf20594d13a5 (127.21.84.135). New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:20.542048 31869 raft_consensus.cc:482] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:20.542268 31869 raft_consensus.cc:504] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:20.542891 31869 leader_election.cc:288] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:20.543303 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95cb5060152e4237988a4c5723a62601" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:20.543447 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95cb5060152e4237988a4c5723a62601" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:20.543568 30936 raft_consensus.cc:2454] T 95cb5060152e4237988a4c5723a62601 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 0.
I20240501 08:12:20.543715 30672 raft_consensus.cc:2454] T 95cb5060152e4237988a4c5723a62601 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 0.
I20240501 08:12:20.544047 27957 leader_election.cc:302] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a29cd590999b4f32849ff5689cd53c11, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:12:20.547685 31869 raft_consensus.cc:2781] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:20.547910 31869 raft_consensus.cc:482] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:20.548022 31869 raft_consensus.cc:3037] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:20.550137 31869 raft_consensus.cc:504] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:20.551088 31869 leader_election.cc:288] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Requested vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:20.551246 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95cb5060152e4237988a4c5723a62601" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:20.551448 30672 raft_consensus.cc:3037] T 95cb5060152e4237988a4c5723a62601 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:20.552107 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95cb5060152e4237988a4c5723a62601" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:20.552899 30936 raft_consensus.cc:3037] T 95cb5060152e4237988a4c5723a62601 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:20.554630 30672 raft_consensus.cc:2454] T 95cb5060152e4237988a4c5723a62601 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 1.
I20240501 08:12:20.555235 27955 leader_election.cc:302] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a022e1a501f4d44a0468054d6c3de47, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:12:20.555737 32062 raft_consensus.cc:2781] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:20.557667 30936 raft_consensus.cc:2454] T 95cb5060152e4237988a4c5723a62601 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 1.
I20240501 08:12:20.562506 32062 raft_consensus.cc:686] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Becoming Leader. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Running, Role: LEADER
I20240501 08:12:20.562878 32062 consensus_queue.cc:237] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:20.567103 27613 catalog_manager.cc:5484] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to aad81047009a4ffcbc450b60030583b5 (127.21.84.131). New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:20.645921 31748 raft_consensus.cc:482] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:20.646171 31748 raft_consensus.cc:504] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:20.646835 31748 leader_election.cc:288] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:20.647356 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8da14ba7c146a9b65d64625ccaf59a" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:12:20.647337 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8da14ba7c146a9b65d64625ccaf59a" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:20.647595 30672 raft_consensus.cc:2454] T df8da14ba7c146a9b65d64625ccaf59a P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:20.648124 28021 raft_consensus.cc:2454] T df8da14ba7c146a9b65d64625ccaf59a P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:20.648188 30737 leader_election.cc:302] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:20.648519 31748 raft_consensus.cc:2781] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:20.648643 31748 raft_consensus.cc:482] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:20.648733 31748 raft_consensus.cc:3037] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:20.650694 31748 raft_consensus.cc:504] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:20.651289 31748 leader_election.cc:288] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Requested vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:20.651739 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8da14ba7c146a9b65d64625ccaf59a" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:20.651803 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8da14ba7c146a9b65d64625ccaf59a" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:12:20.651979 28021 raft_consensus.cc:3037] T df8da14ba7c146a9b65d64625ccaf59a P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:20.651994 30672 raft_consensus.cc:3037] T df8da14ba7c146a9b65d64625ccaf59a P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:20.654994 30672 raft_consensus.cc:2454] T df8da14ba7c146a9b65d64625ccaf59a P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:20.655422 30737 leader_election.cc:302] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:20.655740 31748 raft_consensus.cc:2781] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:20.655880 31748 raft_consensus.cc:686] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Becoming Leader. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Running, Role: LEADER
I20240501 08:12:20.656195 31748 consensus_queue.cc:237] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:20.656613 28021 raft_consensus.cc:2454] T df8da14ba7c146a9b65d64625ccaf59a P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:20.660151 27618 catalog_manager.cc:5484] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64998323a1f4482d8704bf20594d13a5 (127.21.84.135). New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:20.805214 27618 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:20.811744 28021 raft_consensus.cc:1261] T df8da14ba7c146a9b65d64625ccaf59a P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:20.812418 30672 raft_consensus.cc:1261] T df8da14ba7c146a9b65d64625ccaf59a P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:20.812371 32028 consensus_queue.cc:1035] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:20.813084 32028 consensus_queue.cc:1035] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:20.820488 27618 catalog_manager.cc:5576] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305) does not have the latest schema for tablet df8da14ba7c146a9b65d64625ccaf59a (table rebalancer_dynamic_table [id=6c3dec485395467aa63b258afc95201e]). Expected version 1 got 0
I20240501 08:12:20.821954 27613 catalog_manager.cc:5576] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933) does not have the latest schema for tablet df8da14ba7c146a9b65d64625ccaf59a (table rebalancer_dynamic_table [id=6c3dec485395467aa63b258afc95201e]). Expected version 1 got 0
I20240501 08:12:20.824594 32072 tablet.cc:1722] T df8da14ba7c146a9b65d64625ccaf59a P aad81047009a4ffcbc450b60030583b5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:20.825192 32071 tablet.cc:1722] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:20.825357 32072 tablet.cc:1620] T df8da14ba7c146a9b65d64625ccaf59a P aad81047009a4ffcbc450b60030583b5: MemRowSet was empty: no flush needed.
I20240501 08:12:20.825788 32071 tablet.cc:1620] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5: MemRowSet was empty: no flush needed.
I20240501 08:12:20.833768 32074 tablet.cc:1722] T df8da14ba7c146a9b65d64625ccaf59a P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:20.834297 27613 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=6c3dec485395467aa63b258afc95201e] alter complete (version 1)
I20240501 08:12:20.834539 32074 tablet.cc:1620] T df8da14ba7c146a9b65d64625ccaf59a P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:20.845778 27613 catalog_manager.cc:5576] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933) does not have the latest schema for tablet df8da14ba7c146a9b65d64625ccaf59a (table rebalancer_dynamic_table [id=6c3dec485395467aa63b258afc95201e]). Expected version 1 got 0
I20240501 08:12:20.846670 27618 catalog_manager.cc:5576] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305) does not have the latest schema for tablet df8da14ba7c146a9b65d64625ccaf59a (table rebalancer_dynamic_table [id=6c3dec485395467aa63b258afc95201e]). Expected version 1 got 0
I20240501 08:12:20.879329 31921 consensus_queue.cc:1035] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:20.902248 32017 consensus_queue.cc:1035] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 08:12:20.938722 27618 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:20.947338 32071 tablet.cc:1722] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:20.947861 32071 tablet.cc:1620] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5: MemRowSet was empty: no flush needed.
I20240501 08:12:20.950204 32072 tablet.cc:1722] T df8da14ba7c146a9b65d64625ccaf59a P aad81047009a4ffcbc450b60030583b5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:20.950713 32072 tablet.cc:1620] T df8da14ba7c146a9b65d64625ccaf59a P aad81047009a4ffcbc450b60030583b5: MemRowSet was empty: no flush needed.
I20240501 08:12:20.958732 32074 tablet.cc:1722] T df8da14ba7c146a9b65d64625ccaf59a P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:20.959249 32074 tablet.cc:1620] T df8da14ba7c146a9b65d64625ccaf59a P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:20.962308 27613 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=6c3dec485395467aa63b258afc95201e] alter complete (version 2)
I20240501 08:12:21.064231 27613 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:21.066540 32062 consensus_queue.cc:1035] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:21.074134 32071 tablet.cc:1722] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:21.075484 32074 tablet.cc:1722] T df8da14ba7c146a9b65d64625ccaf59a P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:21.075611 32072 tablet.cc:1722] T df8da14ba7c146a9b65d64625ccaf59a P aad81047009a4ffcbc450b60030583b5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:21.085556 32063 consensus_queue.cc:1035] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:21.088960 27613 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table_new_name [id=6c3dec485395467aa63b258afc95201e] alter complete (version 3)
I20240501 08:12:21.094640 27618 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_extra_table_36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:21.097802 32073 consensus_queue.cc:1035] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:21.104799 32028 consensus_queue.cc:1035] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:21.107038 28131 tablet_service.cc:1467] Processing CreateTablet for tablet 893c1555ca0849da8f48a4f253b42153 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=a0611396e39b4372a1b4597945f4a169]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:21.107647 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 893c1555ca0849da8f48a4f253b42153. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.108754 30514 tablet_service.cc:1467] Processing CreateTablet for tablet 893c1555ca0849da8f48a4f253b42153 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=a0611396e39b4372a1b4597945f4a169]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:21.109200 30514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 893c1555ca0849da8f48a4f253b42153. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.111496 30915 tablet_service.cc:1467] Processing CreateTablet for tablet 4e58bc2dffad423f942ce50144030da4 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=a0611396e39b4372a1b4597945f4a169]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:21.112048 30915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e58bc2dffad423f942ce50144030da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.113458 30517 tablet_service.cc:1467] Processing CreateTablet for tablet 4e58bc2dffad423f942ce50144030da4 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=a0611396e39b4372a1b4597945f4a169]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:21.113842 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e58bc2dffad423f942ce50144030da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.114806 30916 tablet_service.cc:1467] Processing CreateTablet for tablet 462c892043d8475c99389c6580a66f0c (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=a0611396e39b4372a1b4597945f4a169]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:21.115233 30916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 462c892043d8475c99389c6580a66f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.120350 30782 tablet_service.cc:1467] Processing CreateTablet for tablet 462c892043d8475c99389c6580a66f0c (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=a0611396e39b4372a1b4597945f4a169]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:21.116854 30784 tablet_service.cc:1467] Processing CreateTablet for tablet 893c1555ca0849da8f48a4f253b42153 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=a0611396e39b4372a1b4597945f4a169]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:21.120820 30782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 462c892043d8475c99389c6580a66f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.122649 30651 tablet_service.cc:1467] Processing CreateTablet for tablet 462c892043d8475c99389c6580a66f0c (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=a0611396e39b4372a1b4597945f4a169]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:21.123598 30651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 462c892043d8475c99389c6580a66f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.124338 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 893c1555ca0849da8f48a4f253b42153. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.123287 30652 tablet_service.cc:1467] Processing CreateTablet for tablet 4e58bc2dffad423f942ce50144030da4 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=a0611396e39b4372a1b4597945f4a169]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:21.144515 32086 tablet_bootstrap.cc:492] T 462c892043d8475c99389c6580a66f0c P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:21.146878 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e58bc2dffad423f942ce50144030da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.147711 32086 tablet_bootstrap.cc:654] T 462c892043d8475c99389c6580a66f0c P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:21.166378 32091 tablet_bootstrap.cc:492] T 893c1555ca0849da8f48a4f253b42153 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:21.176594 32087 tablet_bootstrap.cc:492] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:21.180672 32092 tablet_bootstrap.cc:492] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:21.181282 32089 tablet_bootstrap.cc:492] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:21.183188 32089 tablet_bootstrap.cc:654] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:21.183737 32092 tablet_bootstrap.cc:654] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:21.184723 32087 tablet_bootstrap.cc:654] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:21.190819 32086 tablet_bootstrap.cc:492] T 462c892043d8475c99389c6580a66f0c P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:21.191082 32086 ts_tablet_manager.cc:1367] T 462c892043d8475c99389c6580a66f0c P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.007s
I20240501 08:12:21.191289 32091 tablet_bootstrap.cc:654] T 893c1555ca0849da8f48a4f253b42153 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:21.192087 32086 raft_consensus.cc:348] T 462c892043d8475c99389c6580a66f0c P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:21.192332 32086 raft_consensus.cc:374] T 462c892043d8475c99389c6580a66f0c P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:21.192427 32086 raft_consensus.cc:729] T 462c892043d8475c99389c6580a66f0c P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:21.192713 32092 tablet_bootstrap.cc:492] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:21.192684 32086 consensus_queue.cc:260] T 462c892043d8475c99389c6580a66f0c P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:21.192978 32092 ts_tablet_manager.cc:1367] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20240501 08:12:21.193759 32086 ts_tablet_manager.cc:1398] T 462c892043d8475c99389c6580a66f0c P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:21.194144 32086 tablet_bootstrap.cc:492] T 893c1555ca0849da8f48a4f253b42153 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:21.194468 32092 raft_consensus.cc:348] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:21.194708 32092 raft_consensus.cc:374] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:21.194864 32092 raft_consensus.cc:729] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:21.195111 32092 consensus_queue.cc:260] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:21.196606 32086 tablet_bootstrap.cc:654] T 893c1555ca0849da8f48a4f253b42153 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:21.196867 32092 ts_tablet_manager.cc:1398] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 08:12:21.197187 32092 tablet_bootstrap.cc:492] T 4e58bc2dffad423f942ce50144030da4 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:21.198591 32087 tablet_bootstrap.cc:492] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:21.198781 32087 ts_tablet_manager.cc:1367] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.004s
I20240501 08:12:21.199666 32087 raft_consensus.cc:348] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:21.199903 32087 raft_consensus.cc:374] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:21.200009 32087 raft_consensus.cc:729] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:21.200181 32092 tablet_bootstrap.cc:654] T 4e58bc2dffad423f942ce50144030da4 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:21.201683 32087 consensus_queue.cc:260] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:21.202109 27618 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:21.202306 27618 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:21.203467 32087 ts_tablet_manager.cc:1398] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:12:21.203800 32087 tablet_bootstrap.cc:492] T 462c892043d8475c99389c6580a66f0c P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:21.206869 32087 tablet_bootstrap.cc:654] T 462c892043d8475c99389c6580a66f0c P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:21.213392 32092 tablet_bootstrap.cc:492] T 4e58bc2dffad423f942ce50144030da4 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:21.213742 32092 ts_tablet_manager.cc:1367] T 4e58bc2dffad423f942ce50144030da4 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.007s
I20240501 08:12:21.215461 32092 raft_consensus.cc:348] T 4e58bc2dffad423f942ce50144030da4 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:21.215808 32092 raft_consensus.cc:374] T 4e58bc2dffad423f942ce50144030da4 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:21.215942 32092 raft_consensus.cc:729] T 4e58bc2dffad423f942ce50144030da4 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:21.216282 32092 consensus_queue.cc:260] T 4e58bc2dffad423f942ce50144030da4 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:21.218827 32092 ts_tablet_manager.cc:1398] T 4e58bc2dffad423f942ce50144030da4 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:12:21.222493 27618 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet df8da14ba7c146a9b65d64625ccaf59a
I20240501 08:12:21.223489 30784 tablet_service.cc:1514] Processing DeleteTablet for tablet df8da14ba7c146a9b65d64625ccaf59a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:21 UTC) from {username='slave'} at 127.0.0.1:38952
I20240501 08:12:21.223888 30651 tablet_service.cc:1514] Processing DeleteTablet for tablet df8da14ba7c146a9b65d64625ccaf59a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:21 UTC) from {username='slave'} at 127.0.0.1:57156
I20240501 08:12:21.224256 28002 tablet_service.cc:1514] Processing DeleteTablet for tablet df8da14ba7c146a9b65d64625ccaf59a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:21 UTC) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:21.238034 32101 tablet_replica.cc:330] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5: stopping tablet replica
I20240501 08:12:21.237892 32103 tablet_replica.cc:330] T df8da14ba7c146a9b65d64625ccaf59a P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:21.238534 32101 raft_consensus.cc:2229] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:21.238813 32103 raft_consensus.cc:2229] T df8da14ba7c146a9b65d64625ccaf59a P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:21.239063 32101 raft_consensus.cc:2258] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:21.239588 32103 raft_consensus.cc:2258] T df8da14ba7c146a9b65d64625ccaf59a P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:21.240278 32101 ts_tablet_manager.cc:1875] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:21.242141 32103 ts_tablet_manager.cc:1875] T df8da14ba7c146a9b65d64625ccaf59a P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:21.246959 32102 tablet_replica.cc:330] T df8da14ba7c146a9b65d64625ccaf59a P 9a022e1a501f4d44a0468054d6c3de47: stopping tablet replica
I20240501 08:12:21.248020 32102 raft_consensus.cc:2229] T df8da14ba7c146a9b65d64625ccaf59a P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:21.252207 32102 raft_consensus.cc:2258] T df8da14ba7c146a9b65d64625ccaf59a P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:21.253670 32102 ts_tablet_manager.cc:1875] T df8da14ba7c146a9b65d64625ccaf59a P 9a022e1a501f4d44a0468054d6c3de47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:21.255223 32101 ts_tablet_manager.cc:1888] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:21.255416 32101 log.cc:1196] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal/wals/df8da14ba7c146a9b65d64625ccaf59a
I20240501 08:12:21.256237 32101 ts_tablet_manager.cc:1909] T df8da14ba7c146a9b65d64625ccaf59a P 64998323a1f4482d8704bf20594d13a5: Deleting consensus metadata
I20240501 08:12:21.257179 32089 tablet_bootstrap.cc:492] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:21.257433 27605 catalog_manager.cc:4830] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): tablet df8da14ba7c146a9b65d64625ccaf59a (table rebalancer_dynamic_table_new_name [id=6c3dec485395467aa63b258afc95201e]) successfully deleted
I20240501 08:12:21.257632 32089 ts_tablet_manager.cc:1367] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.076s user 0.027s sys 0.007s
I20240501 08:12:21.258567 32089 raft_consensus.cc:348] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:21.258878 32089 raft_consensus.cc:374] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:21.260250 32086 tablet_bootstrap.cc:492] T 893c1555ca0849da8f48a4f253b42153 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:21.260460 32086 ts_tablet_manager.cc:1367] T 893c1555ca0849da8f48a4f253b42153 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.066s user 0.009s sys 0.015s
I20240501 08:12:21.259022 32089 raft_consensus.cc:729] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:21.261258 32089 consensus_queue.cc:260] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:21.261535 32086 raft_consensus.cc:348] T 893c1555ca0849da8f48a4f253b42153 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:21.261878 32086 raft_consensus.cc:374] T 893c1555ca0849da8f48a4f253b42153 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:21.262050 32086 raft_consensus.cc:729] T 893c1555ca0849da8f48a4f253b42153 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:21.262391 32086 consensus_queue.cc:260] T 893c1555ca0849da8f48a4f253b42153 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:21.263110 32086 ts_tablet_manager.cc:1398] T 893c1555ca0849da8f48a4f253b42153 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 08:12:21.263223 32103 ts_tablet_manager.cc:1888] T df8da14ba7c146a9b65d64625ccaf59a P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:21.263368 32103 log.cc:1196] T df8da14ba7c146a9b65d64625ccaf59a P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/df8da14ba7c146a9b65d64625ccaf59a
I20240501 08:12:21.263849 32103 ts_tablet_manager.cc:1909] T df8da14ba7c146a9b65d64625ccaf59a P aad81047009a4ffcbc450b60030583b5: Deleting consensus metadata
I20240501 08:12:21.265175 32089 ts_tablet_manager.cc:1398] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:21.265499 32089 tablet_bootstrap.cc:492] T 4e58bc2dffad423f942ce50144030da4 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:21.266780 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet df8da14ba7c146a9b65d64625ccaf59a (table rebalancer_dynamic_table_new_name [id=6c3dec485395467aa63b258afc95201e]) successfully deleted
I20240501 08:12:21.267375 32089 tablet_bootstrap.cc:654] T 4e58bc2dffad423f942ce50144030da4 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:21.269603 32102 ts_tablet_manager.cc:1888] T df8da14ba7c146a9b65d64625ccaf59a P 9a022e1a501f4d44a0468054d6c3de47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:21.269857 32102 log.cc:1196] T df8da14ba7c146a9b65d64625ccaf59a P 9a022e1a501f4d44a0468054d6c3de47: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/wals/df8da14ba7c146a9b65d64625ccaf59a
I20240501 08:12:21.270632 32102 ts_tablet_manager.cc:1909] T df8da14ba7c146a9b65d64625ccaf59a P 9a022e1a501f4d44a0468054d6c3de47: Deleting consensus metadata
I20240501 08:12:21.272192 27604 catalog_manager.cc:4830] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): tablet df8da14ba7c146a9b65d64625ccaf59a (table rebalancer_dynamic_table_new_name [id=6c3dec485395467aa63b258afc95201e]) successfully deleted
I20240501 08:12:21.284198 32091 tablet_bootstrap.cc:492] T 893c1555ca0849da8f48a4f253b42153 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:12:21.284381 32091 ts_tablet_manager.cc:1367] T 893c1555ca0849da8f48a4f253b42153 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.118s user 0.007s sys 0.000s
I20240501 08:12:21.284619 32087 tablet_bootstrap.cc:492] T 462c892043d8475c99389c6580a66f0c P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:21.285213 32091 raft_consensus.cc:348] T 893c1555ca0849da8f48a4f253b42153 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:21.285449 32091 raft_consensus.cc:374] T 893c1555ca0849da8f48a4f253b42153 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:21.285552 32091 raft_consensus.cc:729] T 893c1555ca0849da8f48a4f253b42153 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:12:21.285828 32091 consensus_queue.cc:260] T 893c1555ca0849da8f48a4f253b42153 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:21.286198 32087 ts_tablet_manager.cc:1367] T 462c892043d8475c99389c6580a66f0c P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.082s user 0.011s sys 0.031s
I20240501 08:12:21.286671 32091 ts_tablet_manager.cc:1398] T 893c1555ca0849da8f48a4f253b42153 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:21.287374 32087 raft_consensus.cc:348] T 462c892043d8475c99389c6580a66f0c P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:21.287688 32087 raft_consensus.cc:374] T 462c892043d8475c99389c6580a66f0c P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:21.287773 32087 raft_consensus.cc:729] T 462c892043d8475c99389c6580a66f0c P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:21.288261 32087 consensus_queue.cc:260] T 462c892043d8475c99389c6580a66f0c P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:21.289197 32087 ts_tablet_manager.cc:1398] T 462c892043d8475c99389c6580a66f0c P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240501 08:12:21.291714 32089 tablet_bootstrap.cc:492] T 4e58bc2dffad423f942ce50144030da4 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:21.291921 32089 ts_tablet_manager.cc:1367] T 4e58bc2dffad423f942ce50144030da4 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.016s
I20240501 08:12:21.292657 32089 raft_consensus.cc:348] T 4e58bc2dffad423f942ce50144030da4 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:21.292892 32089 raft_consensus.cc:374] T 4e58bc2dffad423f942ce50144030da4 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:21.292966 32089 raft_consensus.cc:729] T 4e58bc2dffad423f942ce50144030da4 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:21.293226 32089 consensus_queue.cc:260] T 4e58bc2dffad423f942ce50144030da4 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:21.293895 32089 ts_tablet_manager.cc:1398] T 4e58bc2dffad423f942ce50144030da4 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:21.309624 32013 raft_consensus.cc:482] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:21.309835 32013 raft_consensus.cc:504] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:21.310521 32013 leader_election.cc:288] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:21.311079 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e58bc2dffad423f942ce50144030da4" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:21.311203 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e58bc2dffad423f942ce50144030da4" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:21.311343 30672 raft_consensus.cc:2454] T 4e58bc2dffad423f942ce50144030da4 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:21.311424 30537 raft_consensus.cc:2454] T 4e58bc2dffad423f942ce50144030da4 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:21.311820 30873 leader_election.cc:302] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:21.312110 32013 raft_consensus.cc:2781] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:21.312238 32013 raft_consensus.cc:482] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:21.312330 32013 raft_consensus.cc:3037] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:21.316403 32013 raft_consensus.cc:504] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:21.317305 32013 leader_election.cc:288] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Requested vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:21.318130 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e58bc2dffad423f942ce50144030da4" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:21.318130 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e58bc2dffad423f942ce50144030da4" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:21.318382 30537 raft_consensus.cc:3037] T 4e58bc2dffad423f942ce50144030da4 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:21.318379 30672 raft_consensus.cc:3037] T 4e58bc2dffad423f942ce50144030da4 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:21.321293 30672 raft_consensus.cc:2454] T 4e58bc2dffad423f942ce50144030da4 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:21.321280 30537 raft_consensus.cc:2454] T 4e58bc2dffad423f942ce50144030da4 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:21.321792 30873 leader_election.cc:302] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:21.322217 32013 raft_consensus.cc:2781] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:21.322347 32013 raft_consensus.cc:686] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Becoming Leader. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:12:21.322605 32013 consensus_queue.cc:237] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:21.325273 27620 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:21.327067 27617 catalog_manager.cc:5484] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: term changed from 0 to 1, leader changed from <none> to a29cd590999b4f32849ff5689cd53c11 (127.21.84.136). New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:21.334929 30784 tablet_service.cc:1467] Processing CreateTablet for tablet 699a7d80c6a846499c078f57847c2e72 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=14d7eec653644cd88d87729a93b628cb]), partition=
I20240501 08:12:21.335191 30915 tablet_service.cc:1467] Processing CreateTablet for tablet 699a7d80c6a846499c078f57847c2e72 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=14d7eec653644cd88d87729a93b628cb]), partition=
I20240501 08:12:21.335362 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 699a7d80c6a846499c078f57847c2e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.335611 30915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 699a7d80c6a846499c078f57847c2e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.340073 32086 tablet_bootstrap.cc:492] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:21.340457 30514 tablet_service.cc:1467] Processing CreateTablet for tablet 699a7d80c6a846499c078f57847c2e72 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=14d7eec653644cd88d87729a93b628cb]), partition=
I20240501 08:12:21.340831 32087 tablet_bootstrap.cc:492] T 699a7d80c6a846499c078f57847c2e72 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:21.340859 30514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 699a7d80c6a846499c078f57847c2e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.342029 32086 tablet_bootstrap.cc:654] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:21.342636 32087 tablet_bootstrap.cc:654] T 699a7d80c6a846499c078f57847c2e72 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:21.345451 32086 tablet_bootstrap.cc:492] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:21.345644 32086 ts_tablet_manager.cc:1367] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240501 08:12:21.346705 32086 raft_consensus.cc:348] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:21.347016 32086 raft_consensus.cc:374] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:21.347100 32092 tablet_bootstrap.cc:492] T 699a7d80c6a846499c078f57847c2e72 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:21.347121 32086 raft_consensus.cc:729] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:21.347534 32086 consensus_queue.cc:260] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:21.348389 32086 ts_tablet_manager.cc:1398] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 08:12:21.348613 32092 tablet_bootstrap.cc:654] T 699a7d80c6a846499c078f57847c2e72 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:21.354104 32087 tablet_bootstrap.cc:492] T 699a7d80c6a846499c078f57847c2e72 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:21.354279 32087 ts_tablet_manager.cc:1367] T 699a7d80c6a846499c078f57847c2e72 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20240501 08:12:21.355357 32092 tablet_bootstrap.cc:492] T 699a7d80c6a846499c078f57847c2e72 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:21.355530 32092 ts_tablet_manager.cc:1367] T 699a7d80c6a846499c078f57847c2e72 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.002s
I20240501 08:12:21.355407 32087 raft_consensus.cc:348] T 699a7d80c6a846499c078f57847c2e72 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:21.355687 32087 raft_consensus.cc:374] T 699a7d80c6a846499c078f57847c2e72 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:21.355794 32087 raft_consensus.cc:729] T 699a7d80c6a846499c078f57847c2e72 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:21.356055 32087 consensus_queue.cc:260] T 699a7d80c6a846499c078f57847c2e72 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:21.356233 32092 raft_consensus.cc:348] T 699a7d80c6a846499c078f57847c2e72 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:21.356426 32092 raft_consensus.cc:374] T 699a7d80c6a846499c078f57847c2e72 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:21.356505 32092 raft_consensus.cc:729] T 699a7d80c6a846499c078f57847c2e72 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:21.356561 32087 ts_tablet_manager.cc:1398] T 699a7d80c6a846499c078f57847c2e72 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:21.356710 32092 consensus_queue.cc:260] T 699a7d80c6a846499c078f57847c2e72 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:21.360656 32092 ts_tablet_manager.cc:1398] T 699a7d80c6a846499c078f57847c2e72 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 08:12:21.376721 31876 raft_consensus.cc:482] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:21.376955 31876 raft_consensus.cc:504] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:21.378620 31876 leader_election.cc:288] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:21.378612 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "462c892043d8475c99389c6580a66f0c" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:21.378855 30804 raft_consensus.cc:2454] T 462c892043d8475c99389c6580a66f0c P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 0.
I20240501 08:12:21.379159 30935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "462c892043d8475c99389c6580a66f0c" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:21.379354 30606 leader_election.cc:302] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:21.379402 30935 raft_consensus.cc:2454] T 462c892043d8475c99389c6580a66f0c P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 0.
I20240501 08:12:21.380034 31876 raft_consensus.cc:2781] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:21.380154 31876 raft_consensus.cc:482] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:21.380239 31876 raft_consensus.cc:3037] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:21.382342 31876 raft_consensus.cc:504] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:21.383082 31876 leader_election.cc:288] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 election: Requested vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:21.383471 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "462c892043d8475c99389c6580a66f0c" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:21.383489 30935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "462c892043d8475c99389c6580a66f0c" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:21.383680 30935 raft_consensus.cc:3037] T 462c892043d8475c99389c6580a66f0c P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:21.383700 30804 raft_consensus.cc:3037] T 462c892043d8475c99389c6580a66f0c P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:21.385735 30935 raft_consensus.cc:2454] T 462c892043d8475c99389c6580a66f0c P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 1.
I20240501 08:12:21.386265 30607 leader_election.cc:302] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a022e1a501f4d44a0468054d6c3de47, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:21.386631 31876 raft_consensus.cc:2781] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:21.387308 30804 raft_consensus.cc:2454] T 462c892043d8475c99389c6580a66f0c P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 1.
I20240501 08:12:21.388154 31876 raft_consensus.cc:686] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Becoming Leader. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Running, Role: LEADER
I20240501 08:12:21.389318 31876 consensus_queue.cc:237] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:21.393299 27620 catalog_manager.cc:5484] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134). New cstate: current_term: 1 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:21.421494 32073 raft_consensus.cc:482] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:21.421713 32073 raft_consensus.cc:504] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:21.422716 32073 leader_election.cc:288] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:21.423624 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "699a7d80c6a846499c078f57847c2e72" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:21.423841 30537 raft_consensus.cc:2454] T 699a7d80c6a846499c078f57847c2e72 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:21.423961 30935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "699a7d80c6a846499c078f57847c2e72" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:21.424224 30935 raft_consensus.cc:2454] T 699a7d80c6a846499c078f57847c2e72 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:21.424324 30740 leader_election.cc:302] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 64998323a1f4482d8704bf20594d13a5; no voters:
I20240501 08:12:21.424672 31748 raft_consensus.cc:2781] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:21.424808 31748 raft_consensus.cc:482] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:21.424906 31748 raft_consensus.cc:3037] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:21.427085 31748 raft_consensus.cc:504] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:21.427778 31748 leader_election.cc:288] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Requested vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:21.428249 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "699a7d80c6a846499c078f57847c2e72" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:21.428475 30537 raft_consensus.cc:3037] T 699a7d80c6a846499c078f57847c2e72 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:21.428655 30935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "699a7d80c6a846499c078f57847c2e72" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:21.428892 30935 raft_consensus.cc:3037] T 699a7d80c6a846499c078f57847c2e72 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:21.431443 30537 raft_consensus.cc:2454] T 699a7d80c6a846499c078f57847c2e72 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:21.432204 30740 leader_election.cc:302] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 64998323a1f4482d8704bf20594d13a5; no voters:
I20240501 08:12:21.432514 31748 raft_consensus.cc:2781] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:21.432652 31748 raft_consensus.cc:686] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Becoming Leader. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Running, Role: LEADER
I20240501 08:12:21.432914 31748 consensus_queue.cc:237] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:21.434641 30935 raft_consensus.cc:2454] T 699a7d80c6a846499c078f57847c2e72 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:21.437445 27617 catalog_manager.cc:5484] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64998323a1f4482d8704bf20594d13a5 (127.21.84.135). New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:21.442399 32111 raft_consensus.cc:482] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:21.442580 32111 raft_consensus.cc:504] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:21.443434 32111 leader_election.cc:288] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:21.445582 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "893c1555ca0849da8f48a4f253b42153" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:12:21.445976 28151 raft_consensus.cc:2454] T 893c1555ca0849da8f48a4f253b42153 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 0.
I20240501 08:12:21.446564 30470 leader_election.cc:302] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:12:21.446988 32111 raft_consensus.cc:2781] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:21.447127 32111 raft_consensus.cc:482] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:21.447245 32111 raft_consensus.cc:3037] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:21.454418 32111 raft_consensus.cc:504] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:21.455554 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "893c1555ca0849da8f48a4f253b42153" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:12:21.455799 28151 raft_consensus.cc:3037] T 893c1555ca0849da8f48a4f253b42153 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:21.459504 32111 leader_election.cc:288] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 election: Requested vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:21.461614 28151 raft_consensus.cc:2454] T 893c1555ca0849da8f48a4f253b42153 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 1.
I20240501 08:12:21.462113 30470 leader_election.cc:302] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:12:21.462477 32111 raft_consensus.cc:2781] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:21.463028 32111 raft_consensus.cc:686] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Becoming Leader. State: Replica: 31811708c6774037b82df44024b229b6, State: Running, Role: LEADER
I20240501 08:12:21.463316 32111 consensus_queue.cc:237] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:21.467727 27617 catalog_manager.cc:5484] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31811708c6774037b82df44024b229b6 (127.21.84.133). New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:21.469326 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "893c1555ca0849da8f48a4f253b42153" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:21.469580 30804 raft_consensus.cc:2454] T 893c1555ca0849da8f48a4f253b42153 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 0.
I20240501 08:12:21.469588 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "893c1555ca0849da8f48a4f253b42153" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:21.470113 30803 raft_consensus.cc:3037] T 893c1555ca0849da8f48a4f253b42153 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:21.472357 30803 raft_consensus.cc:2454] T 893c1555ca0849da8f48a4f253b42153 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 1.
I20240501 08:12:21.560724 27617 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:21.567260 30537 raft_consensus.cc:1261] T 699a7d80c6a846499c078f57847c2e72 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:21.567369 30935 raft_consensus.cc:1261] T 699a7d80c6a846499c078f57847c2e72 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:21.567867 31748 consensus_queue.cc:1035] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:21.568145 32073 consensus_queue.cc:1035] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:21.572361 32071 tablet.cc:1722] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:21.572861 32071 tablet.cc:1620] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5: MemRowSet was empty: no flush needed.
I20240501 08:12:21.573613 27617 catalog_manager.cc:5576] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845) does not have the latest schema for tablet 699a7d80c6a846499c078f57847c2e72 (table rebalancer_dynamic_table [id=14d7eec653644cd88d87729a93b628cb]). Expected version 1 got 0
I20240501 08:12:21.576367 27620 catalog_manager.cc:5576] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503) does not have the latest schema for tablet 699a7d80c6a846499c078f57847c2e72 (table rebalancer_dynamic_table [id=14d7eec653644cd88d87729a93b628cb]). Expected version 1 got 0
I20240501 08:12:21.585026 32121 tablet.cc:1722] T 699a7d80c6a846499c078f57847c2e72 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:21.588104 32121 tablet.cc:1620] T 699a7d80c6a846499c078f57847c2e72 P a29cd590999b4f32849ff5689cd53c11: MemRowSet was empty: no flush needed.
I20240501 08:12:21.590786 27617 catalog_manager.cc:5576] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845) does not have the latest schema for tablet 699a7d80c6a846499c078f57847c2e72 (table rebalancer_dynamic_table [id=14d7eec653644cd88d87729a93b628cb]). Expected version 1 got 0
I20240501 08:12:21.592149 27618 catalog_manager.cc:5576] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293) does not have the latest schema for tablet 699a7d80c6a846499c078f57847c2e72 (table rebalancer_dynamic_table [id=14d7eec653644cd88d87729a93b628cb]). Expected version 1 got 0
I20240501 08:12:21.593588 27620 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=14d7eec653644cd88d87729a93b628cb] alter complete (version 1)
I20240501 08:12:21.594285 32123 tablet.cc:1722] T 699a7d80c6a846499c078f57847c2e72 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:21.594995 32123 tablet.cc:1620] T 699a7d80c6a846499c078f57847c2e72 P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:21.700829 27618 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:21.709611 32071 tablet.cc:1722] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:21.709827 32121 tablet.cc:1722] T 699a7d80c6a846499c078f57847c2e72 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:21.710180 32071 tablet.cc:1620] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5: MemRowSet was empty: no flush needed.
I20240501 08:12:21.710345 32121 tablet.cc:1620] T 699a7d80c6a846499c078f57847c2e72 P a29cd590999b4f32849ff5689cd53c11: MemRowSet was empty: no flush needed.
I20240501 08:12:21.710659 32123 tablet.cc:1722] T 699a7d80c6a846499c078f57847c2e72 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:21.711104 32123 tablet.cc:1620] T 699a7d80c6a846499c078f57847c2e72 P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:21.721298 27620 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=14d7eec653644cd88d87729a93b628cb] alter complete (version 2)
I20240501 08:12:21.794134 31876 consensus_queue.cc:1035] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:21.799238 31876 consensus_queue.cc:1035] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:21.823297 27618 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:21.830379 32071 tablet.cc:1722] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:21.831008 32123 tablet.cc:1722] T 699a7d80c6a846499c078f57847c2e72 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:21.832355 32121 tablet.cc:1722] T 699a7d80c6a846499c078f57847c2e72 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:21.841351 27618 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table_new_name [id=14d7eec653644cd88d87729a93b628cb] alter complete (version 3)
I20240501 08:12:21.843446 32013 consensus_queue.cc:1035] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:21.849773 31641 consensus_queue.cc:1035] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:21.874477 32111 consensus_queue.cc:1035] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:21.879761 32111 consensus_queue.cc:1035] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:21.946322 27618 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:21.946616 27618 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:21.949733 27618 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 699a7d80c6a846499c078f57847c2e72
I20240501 08:12:21.950642 30784 tablet_service.cc:1514] Processing DeleteTablet for tablet 699a7d80c6a846499c078f57847c2e72 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:21 UTC) from {username='slave'} at 127.0.0.1:38952
I20240501 08:12:21.951334 30514 tablet_service.cc:1514] Processing DeleteTablet for tablet 699a7d80c6a846499c078f57847c2e72 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:21 UTC) from {username='slave'} at 127.0.0.1:45078
I20240501 08:12:21.951857 30915 tablet_service.cc:1514] Processing DeleteTablet for tablet 699a7d80c6a846499c078f57847c2e72 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:21 UTC) from {username='slave'} at 127.0.0.1:54036
I20240501 08:12:21.958884 32137 tablet_replica.cc:330] T 699a7d80c6a846499c078f57847c2e72 P 31811708c6774037b82df44024b229b6: stopping tablet replica
I20240501 08:12:21.959240 32137 raft_consensus.cc:2229] T 699a7d80c6a846499c078f57847c2e72 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:21.959483 32137 raft_consensus.cc:2258] T 699a7d80c6a846499c078f57847c2e72 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:21.960882 32137 ts_tablet_manager.cc:1875] T 699a7d80c6a846499c078f57847c2e72 P 31811708c6774037b82df44024b229b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:21.967969 32138 tablet_replica.cc:330] T 699a7d80c6a846499c078f57847c2e72 P a29cd590999b4f32849ff5689cd53c11: stopping tablet replica
I20240501 08:12:21.968046 32137 ts_tablet_manager.cc:1888] T 699a7d80c6a846499c078f57847c2e72 P 31811708c6774037b82df44024b229b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:21.968173 32137 log.cc:1196] T 699a7d80c6a846499c078f57847c2e72 P 31811708c6774037b82df44024b229b6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/wals/699a7d80c6a846499c078f57847c2e72
I20240501 08:12:21.968322 32138 raft_consensus.cc:2229] T 699a7d80c6a846499c078f57847c2e72 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:21.968698 32138 raft_consensus.cc:2258] T 699a7d80c6a846499c078f57847c2e72 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:21.969539 32136 tablet_replica.cc:330] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5: stopping tablet replica
I20240501 08:12:21.969766 32136 raft_consensus.cc:2229] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:21.970139 32138 ts_tablet_manager.cc:1875] T 699a7d80c6a846499c078f57847c2e72 P a29cd590999b4f32849ff5689cd53c11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:21.970324 32136 raft_consensus.cc:2258] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:21.970595 32137 ts_tablet_manager.cc:1909] T 699a7d80c6a846499c078f57847c2e72 P 31811708c6774037b82df44024b229b6: Deleting consensus metadata
I20240501 08:12:21.971163 32136 ts_tablet_manager.cc:1875] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:21.971714 27607 catalog_manager.cc:4830] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): tablet 699a7d80c6a846499c078f57847c2e72 (table rebalancer_dynamic_table_new_name [id=14d7eec653644cd88d87729a93b628cb]) successfully deleted
I20240501 08:12:21.978435 32136 ts_tablet_manager.cc:1888] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:21.978506 32138 ts_tablet_manager.cc:1888] T 699a7d80c6a846499c078f57847c2e72 P a29cd590999b4f32849ff5689cd53c11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:21.978610 32136 log.cc:1196] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal/wals/699a7d80c6a846499c078f57847c2e72
I20240501 08:12:21.978619 32138 log.cc:1196] T 699a7d80c6a846499c078f57847c2e72 P a29cd590999b4f32849ff5689cd53c11: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/wals/699a7d80c6a846499c078f57847c2e72
I20240501 08:12:21.979130 32138 ts_tablet_manager.cc:1909] T 699a7d80c6a846499c078f57847c2e72 P a29cd590999b4f32849ff5689cd53c11: Deleting consensus metadata
I20240501 08:12:21.979138 32136 ts_tablet_manager.cc:1909] T 699a7d80c6a846499c078f57847c2e72 P 64998323a1f4482d8704bf20594d13a5: Deleting consensus metadata
I20240501 08:12:21.980212 27605 catalog_manager.cc:4830] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): tablet 699a7d80c6a846499c078f57847c2e72 (table rebalancer_dynamic_table_new_name [id=14d7eec653644cd88d87729a93b628cb]) successfully deleted
I20240501 08:12:21.980520 27606 catalog_manager.cc:4830] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): tablet 699a7d80c6a846499c078f57847c2e72 (table rebalancer_dynamic_table_new_name [id=14d7eec653644cd88d87729a93b628cb]) successfully deleted
I20240501 08:12:22.027930 27618 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_extra_table_38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:22.037425 30651 tablet_service.cc:1467] Processing CreateTablet for tablet cb6e886e8cdd451d9a192a771482728e (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=099418eaa97444bf9763310daefd6f13]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:22.037958 30651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb6e886e8cdd451d9a192a771482728e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.038012 28131 tablet_service.cc:1467] Processing CreateTablet for tablet cb6e886e8cdd451d9a192a771482728e (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=099418eaa97444bf9763310daefd6f13]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:22.038486 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb6e886e8cdd451d9a192a771482728e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.039278 30784 tablet_service.cc:1467] Processing CreateTablet for tablet b82bf9ba2a7141cc81c6767cdc774c3a (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=099418eaa97444bf9763310daefd6f13]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:22.039680 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82bf9ba2a7141cc81c6767cdc774c3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.042351 27872 tablet_service.cc:1467] Processing CreateTablet for tablet 220add4033694ff381e023324a2a8b02 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=099418eaa97444bf9763310daefd6f13]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:22.042796 27872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 220add4033694ff381e023324a2a8b02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.043795 30514 tablet_service.cc:1467] Processing CreateTablet for tablet cb6e886e8cdd451d9a192a771482728e (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=099418eaa97444bf9763310daefd6f13]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:22.044277 30514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb6e886e8cdd451d9a192a771482728e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.044301 30517 tablet_service.cc:1467] Processing CreateTablet for tablet 220add4033694ff381e023324a2a8b02 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=099418eaa97444bf9763310daefd6f13]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:22.044696 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 220add4033694ff381e023324a2a8b02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.046411 30915 tablet_service.cc:1467] Processing CreateTablet for tablet 220add4033694ff381e023324a2a8b02 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=099418eaa97444bf9763310daefd6f13]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:22.046820 30915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 220add4033694ff381e023324a2a8b02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.048166 27871 tablet_service.cc:1467] Processing CreateTablet for tablet b82bf9ba2a7141cc81c6767cdc774c3a (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=099418eaa97444bf9763310daefd6f13]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:22.048552 27871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82bf9ba2a7141cc81c6767cdc774c3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.052145 30652 tablet_service.cc:1467] Processing CreateTablet for tablet b82bf9ba2a7141cc81c6767cdc774c3a (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=099418eaa97444bf9763310daefd6f13]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:22.052747 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82bf9ba2a7141cc81c6767cdc774c3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.058022 27617 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:22.071875 32140 tablet_bootstrap.cc:492] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:22.072417 28132 tablet_service.cc:1467] Processing CreateTablet for tablet d2bc2991c8944899b7ea13261917ea88 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dd8355484f7c43fa929bd76787555a79]), partition=
I20240501 08:12:22.072782 28132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2bc2991c8944899b7ea13261917ea88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.073220 30650 tablet_service.cc:1467] Processing CreateTablet for tablet d2bc2991c8944899b7ea13261917ea88 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dd8355484f7c43fa929bd76787555a79]), partition=
I20240501 08:12:22.073733 30650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2bc2991c8944899b7ea13261917ea88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.074651 32142 tablet_bootstrap.cc:492] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:22.080703 32142 tablet_bootstrap.cc:654] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:22.082074 32139 tablet_bootstrap.cc:492] T b82bf9ba2a7141cc81c6767cdc774c3a P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:22.084291 32139 tablet_bootstrap.cc:654] T b82bf9ba2a7141cc81c6767cdc774c3a P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:22.080359 30517 tablet_service.cc:1467] Processing CreateTablet for tablet d2bc2991c8944899b7ea13261917ea88 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dd8355484f7c43fa929bd76787555a79]), partition=
I20240501 08:12:22.090729 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2bc2991c8944899b7ea13261917ea88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.098083 32141 tablet_bootstrap.cc:492] T b82bf9ba2a7141cc81c6767cdc774c3a P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:22.099735 32140 tablet_bootstrap.cc:654] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:22.102537 32142 tablet_bootstrap.cc:492] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:22.102787 32142 ts_tablet_manager.cc:1367] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.006s
I20240501 08:12:22.103068 32141 tablet_bootstrap.cc:654] T b82bf9ba2a7141cc81c6767cdc774c3a P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:22.103312 32144 tablet_bootstrap.cc:492] T d2bc2991c8944899b7ea13261917ea88 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:22.103600 32142 raft_consensus.cc:348] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:22.103827 32142 raft_consensus.cc:374] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:22.103924 32142 raft_consensus.cc:729] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:22.104190 32142 consensus_queue.cc:260] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:22.108331 32142 ts_tablet_manager.cc:1398] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:12:22.109618 32144 tablet_bootstrap.cc:654] T d2bc2991c8944899b7ea13261917ea88 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:22.111927 32142 tablet_bootstrap.cc:492] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:22.113852 32139 tablet_bootstrap.cc:492] T b82bf9ba2a7141cc81c6767cdc774c3a P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:22.114154 32139 ts_tablet_manager.cc:1367] T b82bf9ba2a7141cc81c6767cdc774c3a P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.000s
I20240501 08:12:22.115031 32139 raft_consensus.cc:348] T b82bf9ba2a7141cc81c6767cdc774c3a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:22.115300 32139 raft_consensus.cc:374] T b82bf9ba2a7141cc81c6767cdc774c3a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:22.115429 32139 raft_consensus.cc:729] T b82bf9ba2a7141cc81c6767cdc774c3a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:22.115713 32139 consensus_queue.cc:260] T b82bf9ba2a7141cc81c6767cdc774c3a P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:22.117727 32146 tablet_bootstrap.cc:492] T 220add4033694ff381e023324a2a8b02 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:22.117784 32139 ts_tablet_manager.cc:1398] T b82bf9ba2a7141cc81c6767cdc774c3a P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:22.118739 32142 tablet_bootstrap.cc:654] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:22.120110 32146 tablet_bootstrap.cc:654] T 220add4033694ff381e023324a2a8b02 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:22.123813 32141 tablet_bootstrap.cc:492] T b82bf9ba2a7141cc81c6767cdc774c3a P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:12:22.128465 32141 ts_tablet_manager.cc:1367] T b82bf9ba2a7141cc81c6767cdc774c3a P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.008s
I20240501 08:12:22.131233 32141 raft_consensus.cc:348] T b82bf9ba2a7141cc81c6767cdc774c3a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:22.131624 32141 raft_consensus.cc:374] T b82bf9ba2a7141cc81c6767cdc774c3a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:22.131621 32140 tablet_bootstrap.cc:492] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:22.131752 32141 raft_consensus.cc:729] T b82bf9ba2a7141cc81c6767cdc774c3a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:12:22.131817 32140 ts_tablet_manager.cc:1367] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.063s user 0.013s sys 0.006s
I20240501 08:12:22.132104 32141 consensus_queue.cc:260] T b82bf9ba2a7141cc81c6767cdc774c3a P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:22.132905 32141 ts_tablet_manager.cc:1398] T b82bf9ba2a7141cc81c6767cdc774c3a P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:12:22.133224 32141 tablet_bootstrap.cc:492] T 220add4033694ff381e023324a2a8b02 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:22.137785 32144 tablet_bootstrap.cc:492] T d2bc2991c8944899b7ea13261917ea88 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:12:22.137876 32140 raft_consensus.cc:348] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:22.138180 32144 ts_tablet_manager.cc:1367] T d2bc2991c8944899b7ea13261917ea88 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.035s user 0.007s sys 0.000s
I20240501 08:12:22.138181 32140 raft_consensus.cc:374] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:22.138278 32140 raft_consensus.cc:729] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:22.138777 32141 tablet_bootstrap.cc:654] T 220add4033694ff381e023324a2a8b02 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:22.138671 32140 consensus_queue.cc:260] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:22.138823 32146 tablet_bootstrap.cc:492] T 220add4033694ff381e023324a2a8b02 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:22.138988 32146 ts_tablet_manager.cc:1367] T 220add4033694ff381e023324a2a8b02 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.004s
I20240501 08:12:22.139043 32144 raft_consensus.cc:348] T d2bc2991c8944899b7ea13261917ea88 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:22.139334 32144 raft_consensus.cc:374] T d2bc2991c8944899b7ea13261917ea88 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:22.139472 32144 raft_consensus.cc:729] T d2bc2991c8944899b7ea13261917ea88 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:12:22.139783 32146 raft_consensus.cc:348] T 220add4033694ff381e023324a2a8b02 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:22.139873 32144 consensus_queue.cc:260] T d2bc2991c8944899b7ea13261917ea88 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:22.140050 32146 raft_consensus.cc:374] T 220add4033694ff381e023324a2a8b02 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:22.140177 32146 raft_consensus.cc:729] T 220add4033694ff381e023324a2a8b02 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:22.140453 32146 consensus_queue.cc:260] T 220add4033694ff381e023324a2a8b02 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:22.141170 32146 ts_tablet_manager.cc:1398] T 220add4033694ff381e023324a2a8b02 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:22.142796 32142 tablet_bootstrap.cc:492] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:22.143028 32142 ts_tablet_manager.cc:1367] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.004s
I20240501 08:12:22.143126 32144 ts_tablet_manager.cc:1398] T d2bc2991c8944899b7ea13261917ea88 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:12:22.143504 32144 tablet_bootstrap.cc:492] T cb6e886e8cdd451d9a192a771482728e P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:22.143795 32142 raft_consensus.cc:348] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:22.144081 32142 raft_consensus.cc:374] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:22.144183 32142 raft_consensus.cc:729] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:22.144474 32142 consensus_queue.cc:260] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:22.152355 32140 ts_tablet_manager.cc:1398] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240501 08:12:22.153354 32141 tablet_bootstrap.cc:492] T 220add4033694ff381e023324a2a8b02 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:12:22.153635 32141 ts_tablet_manager.cc:1367] T 220add4033694ff381e023324a2a8b02 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.006s
I20240501 08:12:22.154497 32141 raft_consensus.cc:348] T 220add4033694ff381e023324a2a8b02 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:22.154950 32141 raft_consensus.cc:374] T 220add4033694ff381e023324a2a8b02 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:22.155079 32141 raft_consensus.cc:729] T 220add4033694ff381e023324a2a8b02 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:12:22.155416 32141 consensus_queue.cc:260] T 220add4033694ff381e023324a2a8b02 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:22.156138 32141 ts_tablet_manager.cc:1398] T 220add4033694ff381e023324a2a8b02 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:22.149706 32142 ts_tablet_manager.cc:1398] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240501 08:12:22.156991 32142 tablet_bootstrap.cc:492] T b82bf9ba2a7141cc81c6767cdc774c3a P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:22.157436 32140 tablet_bootstrap.cc:492] T cb6e886e8cdd451d9a192a771482728e P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:22.155431 32144 tablet_bootstrap.cc:654] T cb6e886e8cdd451d9a192a771482728e P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:22.160763 32140 tablet_bootstrap.cc:654] T cb6e886e8cdd451d9a192a771482728e P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:22.159775 32142 tablet_bootstrap.cc:654] T b82bf9ba2a7141cc81c6767cdc774c3a P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:22.171823 32140 tablet_bootstrap.cc:492] T cb6e886e8cdd451d9a192a771482728e P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:22.171823 32144 tablet_bootstrap.cc:492] T cb6e886e8cdd451d9a192a771482728e P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:12:22.172031 32144 ts_tablet_manager.cc:1367] T cb6e886e8cdd451d9a192a771482728e P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.007s
I20240501 08:12:22.173000 32144 raft_consensus.cc:348] T cb6e886e8cdd451d9a192a771482728e P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:22.173233 32144 raft_consensus.cc:374] T cb6e886e8cdd451d9a192a771482728e P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:22.173324 32144 raft_consensus.cc:729] T cb6e886e8cdd451d9a192a771482728e P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:12:22.173563 32144 consensus_queue.cc:260] T cb6e886e8cdd451d9a192a771482728e P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:22.174109 32140 ts_tablet_manager.cc:1367] T cb6e886e8cdd451d9a192a771482728e P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.003s
I20240501 08:12:22.175484 32144 ts_tablet_manager.cc:1398] T cb6e886e8cdd451d9a192a771482728e P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:12:22.176080 32142 tablet_bootstrap.cc:492] T b82bf9ba2a7141cc81c6767cdc774c3a P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:22.176237 32142 ts_tablet_manager.cc:1367] T b82bf9ba2a7141cc81c6767cdc774c3a P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.005s
I20240501 08:12:22.177196 32142 raft_consensus.cc:348] T b82bf9ba2a7141cc81c6767cdc774c3a P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:22.177404 32142 raft_consensus.cc:374] T b82bf9ba2a7141cc81c6767cdc774c3a P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:22.177507 32142 raft_consensus.cc:729] T b82bf9ba2a7141cc81c6767cdc774c3a P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:22.175046 32140 raft_consensus.cc:348] T cb6e886e8cdd451d9a192a771482728e P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:22.177757 32142 consensus_queue.cc:260] T b82bf9ba2a7141cc81c6767cdc774c3a P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:22.177897 32140 raft_consensus.cc:374] T cb6e886e8cdd451d9a192a771482728e P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:22.178040 32140 raft_consensus.cc:729] T cb6e886e8cdd451d9a192a771482728e P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:22.178376 32140 consensus_queue.cc:260] T cb6e886e8cdd451d9a192a771482728e P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:22.178925 32142 ts_tablet_manager.cc:1398] T b82bf9ba2a7141cc81c6767cdc774c3a P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:22.179991 32140 ts_tablet_manager.cc:1398] T cb6e886e8cdd451d9a192a771482728e P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:12:22.180362 32140 tablet_bootstrap.cc:492] T d2bc2991c8944899b7ea13261917ea88 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:22.184315 32140 tablet_bootstrap.cc:654] T d2bc2991c8944899b7ea13261917ea88 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:22.186336 32140 tablet_bootstrap.cc:492] T d2bc2991c8944899b7ea13261917ea88 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:22.186547 32140 ts_tablet_manager.cc:1367] T d2bc2991c8944899b7ea13261917ea88 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 08:12:22.187263 32140 raft_consensus.cc:348] T d2bc2991c8944899b7ea13261917ea88 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:22.187604 32140 raft_consensus.cc:374] T d2bc2991c8944899b7ea13261917ea88 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:22.187708 32140 raft_consensus.cc:729] T d2bc2991c8944899b7ea13261917ea88 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:22.187938 32140 consensus_queue.cc:260] T d2bc2991c8944899b7ea13261917ea88 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:22.188372 32140 ts_tablet_manager.cc:1398] T d2bc2991c8944899b7ea13261917ea88 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:22.248544 31876 raft_consensus.cc:482] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:22.248775 31876 raft_consensus.cc:504] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:22.249629 31876 leader_election.cc:288] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:22.250232 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2bc2991c8944899b7ea13261917ea88" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:12:22.250490 28151 raft_consensus.cc:2454] T d2bc2991c8944899b7ea13261917ea88 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 0.
I20240501 08:12:22.250401 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2bc2991c8944899b7ea13261917ea88" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:22.250746 30536 raft_consensus.cc:2454] T d2bc2991c8944899b7ea13261917ea88 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 0.
I20240501 08:12:22.251358 30605 leader_election.cc:302] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:22.252440 31876 raft_consensus.cc:2781] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:22.252530 31876 raft_consensus.cc:482] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:22.252610 31876 raft_consensus.cc:3037] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:22.255383 31876 raft_consensus.cc:504] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:22.256467 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2bc2991c8944899b7ea13261917ea88" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:12:22.256621 31876 leader_election.cc:288] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 election: Requested vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:22.256672 28151 raft_consensus.cc:3037] T d2bc2991c8944899b7ea13261917ea88 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:22.256865 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2bc2991c8944899b7ea13261917ea88" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:22.257069 30536 raft_consensus.cc:3037] T d2bc2991c8944899b7ea13261917ea88 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:22.261055 28151 raft_consensus.cc:2454] T d2bc2991c8944899b7ea13261917ea88 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 1.
I20240501 08:12:22.261961 30536 raft_consensus.cc:2454] T d2bc2991c8944899b7ea13261917ea88 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 1.
I20240501 08:12:22.262243 30605 leader_election.cc:302] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:22.262935 31876 raft_consensus.cc:2781] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:22.263499 31876 raft_consensus.cc:686] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Becoming Leader. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Running, Role: LEADER
I20240501 08:12:22.263823 31876 consensus_queue.cc:237] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:22.267977 27613 catalog_manager.cc:5484] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134). New cstate: current_term: 1 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:22.299167 31876 raft_consensus.cc:482] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:22.299371 31876 raft_consensus.cc:504] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:22.300218 31876 leader_election.cc:288] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:22.300535 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb6e886e8cdd451d9a192a771482728e" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:12:22.300644 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb6e886e8cdd451d9a192a771482728e" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:22.300762 28151 raft_consensus.cc:2454] T cb6e886e8cdd451d9a192a771482728e P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 0.
I20240501 08:12:22.300922 30536 raft_consensus.cc:2454] T cb6e886e8cdd451d9a192a771482728e P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 0.
I20240501 08:12:22.301122 30605 leader_election.cc:302] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:22.301391 31876 raft_consensus.cc:2781] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:22.301510 31876 raft_consensus.cc:482] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:22.301595 31876 raft_consensus.cc:3037] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:22.303563 31876 raft_consensus.cc:504] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:22.304139 31876 leader_election.cc:288] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 election: Requested vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:22.304523 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb6e886e8cdd451d9a192a771482728e" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:22.304595 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb6e886e8cdd451d9a192a771482728e" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:12:22.304745 30536 raft_consensus.cc:3037] T cb6e886e8cdd451d9a192a771482728e P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:22.304777 28151 raft_consensus.cc:3037] T cb6e886e8cdd451d9a192a771482728e P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:22.306915 30536 raft_consensus.cc:2454] T cb6e886e8cdd451d9a192a771482728e P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 1.
I20240501 08:12:22.306974 28151 raft_consensus.cc:2454] T cb6e886e8cdd451d9a192a771482728e P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 1.
I20240501 08:12:22.307564 30605 leader_election.cc:302] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:22.308046 32112 raft_consensus.cc:2781] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:22.308255 32112 raft_consensus.cc:686] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Becoming Leader. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Running, Role: LEADER
I20240501 08:12:22.308573 32112 consensus_queue.cc:237] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:22.313881 27613 catalog_manager.cc:5484] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134). New cstate: current_term: 1 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:22.330325 32079 raft_consensus.cc:482] T b82bf9ba2a7141cc81c6767cdc774c3a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:22.330555 32079 raft_consensus.cc:504] T b82bf9ba2a7141cc81c6767cdc774c3a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:22.331440 32079 leader_election.cc:288] T b82bf9ba2a7141cc81c6767cdc774c3a P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:22.331936 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82bf9ba2a7141cc81c6767cdc774c3a" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:22.331935 30671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82bf9ba2a7141cc81c6767cdc774c3a" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:22.332176 30671 raft_consensus.cc:2454] T b82bf9ba2a7141cc81c6767cdc774c3a P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:12:22.332159 30804 raft_consensus.cc:2454] T b82bf9ba2a7141cc81c6767cdc774c3a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:12:22.332669 27826 leader_election.cc:302] T b82bf9ba2a7141cc81c6767cdc774c3a P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:22.333038 32079 raft_consensus.cc:2781] T b82bf9ba2a7141cc81c6767cdc774c3a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:22.333151 32079 raft_consensus.cc:482] T b82bf9ba2a7141cc81c6767cdc774c3a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:22.333240 32079 raft_consensus.cc:3037] T b82bf9ba2a7141cc81c6767cdc774c3a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:22.335708 32079 raft_consensus.cc:504] T b82bf9ba2a7141cc81c6767cdc774c3a P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:22.336293 32079 leader_election.cc:288] T b82bf9ba2a7141cc81c6767cdc774c3a P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Requested vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:22.336845 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82bf9ba2a7141cc81c6767cdc774c3a" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:22.336844 30671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82bf9ba2a7141cc81c6767cdc774c3a" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:22.337080 30804 raft_consensus.cc:3037] T b82bf9ba2a7141cc81c6767cdc774c3a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:22.337093 30671 raft_consensus.cc:3037] T b82bf9ba2a7141cc81c6767cdc774c3a P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:22.339943 30804 raft_consensus.cc:2454] T b82bf9ba2a7141cc81c6767cdc774c3a P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:12:22.339931 30671 raft_consensus.cc:2454] T b82bf9ba2a7141cc81c6767cdc774c3a P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:12:22.340386 27825 leader_election.cc:302] T b82bf9ba2a7141cc81c6767cdc774c3a P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a022e1a501f4d44a0468054d6c3de47, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:22.340700 32079 raft_consensus.cc:2781] T b82bf9ba2a7141cc81c6767cdc774c3a P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:22.340873 32079 raft_consensus.cc:686] T b82bf9ba2a7141cc81c6767cdc774c3a P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Becoming Leader. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Running, Role: LEADER
I20240501 08:12:22.341197 32079 consensus_queue.cc:237] T b82bf9ba2a7141cc81c6767cdc774c3a P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:22.345060 27613 catalog_manager.cc:5484] T b82bf9ba2a7141cc81c6767cdc774c3a P f1525cbc82a24d13b65f136695c903af reported cstate change: term changed from 0 to 1, leader changed from <none> to f1525cbc82a24d13b65f136695c903af (127.21.84.130). New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:22.385440 32162 raft_consensus.cc:482] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:22.386083 32162 raft_consensus.cc:504] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:22.387068 32162 leader_election.cc:288] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:12:22.387533 30935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220add4033694ff381e023324a2a8b02" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:22.387624 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220add4033694ff381e023324a2a8b02" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:12:22.387809 30935 raft_consensus.cc:2454] T 220add4033694ff381e023324a2a8b02 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 0.
I20240501 08:12:22.387827 27890 raft_consensus.cc:2454] T 220add4033694ff381e023324a2a8b02 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 0.
I20240501 08:12:22.388268 30473 leader_election.cc:302] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:22.388557 32162 raft_consensus.cc:2781] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:22.388674 32162 raft_consensus.cc:482] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:22.388760 32162 raft_consensus.cc:3037] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:22.391602 32162 raft_consensus.cc:504] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:22.392217 32162 leader_election.cc:288] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 election: Requested vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:12:22.392714 30935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220add4033694ff381e023324a2a8b02" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:22.392937 30935 raft_consensus.cc:3037] T 220add4033694ff381e023324a2a8b02 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:22.395618 30935 raft_consensus.cc:2454] T 220add4033694ff381e023324a2a8b02 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 1.
I20240501 08:12:22.395992 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220add4033694ff381e023324a2a8b02" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:12:22.396205 27890 raft_consensus.cc:3037] T 220add4033694ff381e023324a2a8b02 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:22.398892 27890 raft_consensus.cc:2454] T 220add4033694ff381e023324a2a8b02 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 1.
I20240501 08:12:22.399343 30473 leader_election.cc:302] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:22.399646 32162 raft_consensus.cc:2781] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:22.399878 32162 raft_consensus.cc:686] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Becoming Leader. State: Replica: 31811708c6774037b82df44024b229b6, State: Running, Role: LEADER
I20240501 08:12:22.400177 32162 consensus_queue.cc:237] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:22.401319 27613 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:22.404651 27618 catalog_manager.cc:5484] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31811708c6774037b82df44024b229b6 (127.21.84.133). New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:22.410200 30536 raft_consensus.cc:1261] T d2bc2991c8944899b7ea13261917ea88 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:22.410933 32159 consensus_queue.cc:1035] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:22.411146 28151 raft_consensus.cc:1261] T d2bc2991c8944899b7ea13261917ea88 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:22.411621 32159 consensus_queue.cc:1035] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:22.416409 27618 catalog_manager.cc:5576] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293) does not have the latest schema for tablet d2bc2991c8944899b7ea13261917ea88 (table rebalancer_dynamic_table [id=dd8355484f7c43fa929bd76787555a79]). Expected version 1 got 0
I20240501 08:12:22.417701 27613 catalog_manager.cc:5576] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323) does not have the latest schema for tablet d2bc2991c8944899b7ea13261917ea88 (table rebalancer_dynamic_table [id=dd8355484f7c43fa929bd76787555a79]). Expected version 1 got 0
I20240501 08:12:22.418372 32170 tablet.cc:1722] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:22.419090 32170 tablet.cc:1620] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:22.419803 32171 tablet.cc:1722] T d2bc2991c8944899b7ea13261917ea88 P 91d11eae5ef6432ca86eb8ed446b6ea6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:22.419932 32172 tablet.cc:1722] T d2bc2991c8944899b7ea13261917ea88 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:22.420507 32172 tablet.cc:1620] T d2bc2991c8944899b7ea13261917ea88 P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:22.420697 32171 tablet.cc:1620] T d2bc2991c8944899b7ea13261917ea88 P 91d11eae5ef6432ca86eb8ed446b6ea6: MemRowSet was empty: no flush needed.
I20240501 08:12:22.423269 27613 catalog_manager.cc:5576] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323) does not have the latest schema for tablet d2bc2991c8944899b7ea13261917ea88 (table rebalancer_dynamic_table [id=dd8355484f7c43fa929bd76787555a79]). Expected version 1 got 0
I20240501 08:12:22.425117 27617 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=dd8355484f7c43fa929bd76787555a79] alter complete (version 1)
I20240501 08:12:22.530336 27617 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:22.538043 32170 tablet.cc:1722] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:22.538525 32172 tablet.cc:1722] T d2bc2991c8944899b7ea13261917ea88 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:22.538617 32171 tablet.cc:1722] T d2bc2991c8944899b7ea13261917ea88 P 91d11eae5ef6432ca86eb8ed446b6ea6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:22.538825 32170 tablet.cc:1620] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:22.538985 32172 tablet.cc:1620] T d2bc2991c8944899b7ea13261917ea88 P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:22.539049 32171 tablet.cc:1620] T d2bc2991c8944899b7ea13261917ea88 P 91d11eae5ef6432ca86eb8ed446b6ea6: MemRowSet was empty: no flush needed.
I20240501 08:12:22.548349 27617 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=dd8355484f7c43fa929bd76787555a79] alter complete (version 2)
I20240501 08:12:22.654356 27618 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:22.661535 32170 tablet.cc:1722] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:22.662132 32171 tablet.cc:1722] T d2bc2991c8944899b7ea13261917ea88 P 91d11eae5ef6432ca86eb8ed446b6ea6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:22.662781 32172 tablet.cc:1722] T d2bc2991c8944899b7ea13261917ea88 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:22.672021 27613 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table_new_name [id=dd8355484f7c43fa929bd76787555a79] alter complete (version 3)
I20240501 08:12:22.775419 27617 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:22.775668 27617 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:22.778913 27617 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet d2bc2991c8944899b7ea13261917ea88
I20240501 08:12:22.779948 30517 tablet_service.cc:1514] Processing DeleteTablet for tablet d2bc2991c8944899b7ea13261917ea88 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:22 UTC) from {username='slave'} at 127.0.0.1:45078
I20240501 08:12:22.779955 30652 tablet_service.cc:1514] Processing DeleteTablet for tablet d2bc2991c8944899b7ea13261917ea88 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:22 UTC) from {username='slave'} at 127.0.0.1:57156
I20240501 08:12:22.780227 28132 tablet_service.cc:1514] Processing DeleteTablet for tablet d2bc2991c8944899b7ea13261917ea88 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:22 UTC) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:22.795845 32173 tablet_replica.cc:330] T d2bc2991c8944899b7ea13261917ea88 P 31811708c6774037b82df44024b229b6: stopping tablet replica
I20240501 08:12:22.796157 32173 raft_consensus.cc:2229] T d2bc2991c8944899b7ea13261917ea88 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:22.796465 32173 raft_consensus.cc:2258] T d2bc2991c8944899b7ea13261917ea88 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:22.797634 32078 consensus_queue.cc:1035] T b82bf9ba2a7141cc81c6767cdc774c3a P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:22.797874 32173 ts_tablet_manager.cc:1875] T d2bc2991c8944899b7ea13261917ea88 P 31811708c6774037b82df44024b229b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:22.798978 32175 tablet_replica.cc:330] T d2bc2991c8944899b7ea13261917ea88 P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:22.799366 32175 raft_consensus.cc:2229] T d2bc2991c8944899b7ea13261917ea88 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:22.799801 32175 raft_consensus.cc:2258] T d2bc2991c8944899b7ea13261917ea88 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:22.801198 32175 ts_tablet_manager.cc:1875] T d2bc2991c8944899b7ea13261917ea88 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:22.805141 32174 tablet_replica.cc:330] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47: stopping tablet replica
I20240501 08:12:22.805485 32174 raft_consensus.cc:2229] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:22.806027 32174 raft_consensus.cc:2258] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:22.806126 32173 ts_tablet_manager.cc:1888] T d2bc2991c8944899b7ea13261917ea88 P 31811708c6774037b82df44024b229b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:22.806344 32173 log.cc:1196] T d2bc2991c8944899b7ea13261917ea88 P 31811708c6774037b82df44024b229b6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/wals/d2bc2991c8944899b7ea13261917ea88
I20240501 08:12:22.806986 32173 ts_tablet_manager.cc:1909] T d2bc2991c8944899b7ea13261917ea88 P 31811708c6774037b82df44024b229b6: Deleting consensus metadata
I20240501 08:12:22.807060 32174 ts_tablet_manager.cc:1875] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:22.808540 27607 catalog_manager.cc:4830] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): tablet d2bc2991c8944899b7ea13261917ea88 (table rebalancer_dynamic_table_new_name [id=dd8355484f7c43fa929bd76787555a79]) successfully deleted
I20240501 08:12:22.816210 32174 ts_tablet_manager.cc:1888] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:22.816452 32174 log.cc:1196] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/wals/d2bc2991c8944899b7ea13261917ea88
I20240501 08:12:22.810102 32175 ts_tablet_manager.cc:1888] T d2bc2991c8944899b7ea13261917ea88 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:22.817180 32174 ts_tablet_manager.cc:1909] T d2bc2991c8944899b7ea13261917ea88 P 9a022e1a501f4d44a0468054d6c3de47: Deleting consensus metadata
I20240501 08:12:22.817181 32175 log.cc:1196] T d2bc2991c8944899b7ea13261917ea88 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/d2bc2991c8944899b7ea13261917ea88
I20240501 08:12:22.818428 32175 ts_tablet_manager.cc:1909] T d2bc2991c8944899b7ea13261917ea88 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting consensus metadata
I20240501 08:12:22.818655 27604 catalog_manager.cc:4830] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): tablet d2bc2991c8944899b7ea13261917ea88 (table rebalancer_dynamic_table_new_name [id=dd8355484f7c43fa929bd76787555a79]) successfully deleted
I20240501 08:12:22.820814 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet d2bc2991c8944899b7ea13261917ea88 (table rebalancer_dynamic_table_new_name [id=dd8355484f7c43fa929bd76787555a79]) successfully deleted
I20240501 08:12:22.823064 32078 consensus_queue.cc:1035] T b82bf9ba2a7141cc81c6767cdc774c3a P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:22.839828 32162 consensus_queue.cc:1035] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:22.845005 32162 consensus_queue.cc:1035] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:22.881213 27613 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:22.888375 30652 tablet_service.cc:1467] Processing CreateTablet for tablet 42662587e09e4c98b1830e4f650a4079 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e04c4dccf30b4007a005b103ee9e5531]), partition=
I20240501 08:12:22.888706 28002 tablet_service.cc:1467] Processing CreateTablet for tablet 42662587e09e4c98b1830e4f650a4079 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e04c4dccf30b4007a005b103ee9e5531]), partition=
I20240501 08:12:22.888769 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42662587e09e4c98b1830e4f650a4079. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.888931 30784 tablet_service.cc:1467] Processing CreateTablet for tablet 42662587e09e4c98b1830e4f650a4079 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e04c4dccf30b4007a005b103ee9e5531]), partition=
I20240501 08:12:22.889089 28002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42662587e09e4c98b1830e4f650a4079. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.889299 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42662587e09e4c98b1830e4f650a4079. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.894623 32186 tablet_bootstrap.cc:492] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:22.896842 31876 consensus_queue.cc:1035] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:22.897416 32186 tablet_bootstrap.cc:654] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:22.911443 32159 consensus_queue.cc:1035] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:22.913655 32186 tablet_bootstrap.cc:492] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:22.913852 32186 ts_tablet_manager.cc:1367] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.004s
I20240501 08:12:22.923985 32190 tablet_bootstrap.cc:492] T 42662587e09e4c98b1830e4f650a4079 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:22.927415 32190 tablet_bootstrap.cc:654] T 42662587e09e4c98b1830e4f650a4079 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:22.941845 32186 raft_consensus.cc:348] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:22.942835 32186 raft_consensus.cc:374] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:22.943076 32186 raft_consensus.cc:729] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:22.943544 32186 consensus_queue.cc:260] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:22.944327 32186 ts_tablet_manager.cc:1398] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.030s user 0.001s sys 0.001s
I20240501 08:12:22.947574 32187 tablet_bootstrap.cc:492] T 42662587e09e4c98b1830e4f650a4079 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:22.949970 32187 tablet_bootstrap.cc:654] T 42662587e09e4c98b1830e4f650a4079 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:22.952286 32190 tablet_bootstrap.cc:492] T 42662587e09e4c98b1830e4f650a4079 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:12:22.952445 32190 ts_tablet_manager.cc:1367] T 42662587e09e4c98b1830e4f650a4079 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.003s
I20240501 08:12:22.953049 32190 raft_consensus.cc:348] T 42662587e09e4c98b1830e4f650a4079 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:22.953228 32190 raft_consensus.cc:374] T 42662587e09e4c98b1830e4f650a4079 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:22.953318 32190 raft_consensus.cc:729] T 42662587e09e4c98b1830e4f650a4079 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:12:22.953552 32190 consensus_queue.cc:260] T 42662587e09e4c98b1830e4f650a4079 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:22.954100 32190 ts_tablet_manager.cc:1398] T 42662587e09e4c98b1830e4f650a4079 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:22.954867 32187 tablet_bootstrap.cc:492] T 42662587e09e4c98b1830e4f650a4079 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:22.955013 32187 ts_tablet_manager.cc:1367] T 42662587e09e4c98b1830e4f650a4079 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20240501 08:12:22.955708 32187 raft_consensus.cc:348] T 42662587e09e4c98b1830e4f650a4079 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:22.955938 32187 raft_consensus.cc:374] T 42662587e09e4c98b1830e4f650a4079 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:22.956034 32187 raft_consensus.cc:729] T 42662587e09e4c98b1830e4f650a4079 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:22.956537 32187 consensus_queue.cc:260] T 42662587e09e4c98b1830e4f650a4079 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:22.957216 32187 ts_tablet_manager.cc:1398] T 42662587e09e4c98b1830e4f650a4079 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:22.973500 32159 raft_consensus.cc:482] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:22.973696 32159 raft_consensus.cc:504] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:22.974426 32159 leader_election.cc:288] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:22.974931 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42662587e09e4c98b1830e4f650a4079" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:22.975003 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42662587e09e4c98b1830e4f650a4079" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:12:22.975157 30803 raft_consensus.cc:2454] T 42662587e09e4c98b1830e4f650a4079 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 0.
I20240501 08:12:22.975234 28021 raft_consensus.cc:2454] T 42662587e09e4c98b1830e4f650a4079 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 0.
I20240501 08:12:22.975557 30606 leader_election.cc:302] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:22.976517 32159 raft_consensus.cc:2781] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:22.976609 32159 raft_consensus.cc:482] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:22.976737 32159 raft_consensus.cc:3037] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:22.977149 27613 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_extra_table_40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:22.979261 32159 raft_consensus.cc:504] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:22.979924 32159 leader_election.cc:288] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 election: Requested vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:22.980363 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42662587e09e4c98b1830e4f650a4079" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:22.980576 30803 raft_consensus.cc:3037] T 42662587e09e4c98b1830e4f650a4079 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:22.981240 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42662587e09e4c98b1830e4f650a4079" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:12:22.981439 28021 raft_consensus.cc:3037] T 42662587e09e4c98b1830e4f650a4079 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:22.983115 30803 raft_consensus.cc:2454] T 42662587e09e4c98b1830e4f650a4079 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 1.
I20240501 08:12:22.983489 28021 raft_consensus.cc:2454] T 42662587e09e4c98b1830e4f650a4079 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 1.
I20240501 08:12:22.983556 30606 leader_election.cc:302] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:22.983922 32159 raft_consensus.cc:2781] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:22.984216 32159 raft_consensus.cc:686] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Becoming Leader. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Running, Role: LEADER
I20240501 08:12:22.984580 32159 consensus_queue.cc:237] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:22.988605 30915 tablet_service.cc:1467] Processing CreateTablet for tablet 87804550560d45269e6d4d6fe8938f55 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b218cf6d6a9e4117abbfea2879ddcd54]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:22.989066 30915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87804550560d45269e6d4d6fe8938f55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.989209 30782 tablet_service.cc:1467] Processing CreateTablet for tablet 9b5b0b1e46d14d78806859541e0ab3b6 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b218cf6d6a9e4117abbfea2879ddcd54]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:22.989603 30782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b5b0b1e46d14d78806859541e0ab3b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.990316 28132 tablet_service.cc:1467] Processing CreateTablet for tablet b605254185d140d8ab90fcaa5229f168 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b218cf6d6a9e4117abbfea2879ddcd54]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:22.990327 27871 tablet_service.cc:1467] Processing CreateTablet for tablet 87804550560d45269e6d4d6fe8938f55 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b218cf6d6a9e4117abbfea2879ddcd54]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:22.990722 27871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87804550560d45269e6d4d6fe8938f55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.990758 28132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b605254185d140d8ab90fcaa5229f168. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.991048 27613 catalog_manager.cc:5484] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134). New cstate: current_term: 1 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:22.994843 30517 tablet_service.cc:1467] Processing CreateTablet for tablet 9b5b0b1e46d14d78806859541e0ab3b6 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b218cf6d6a9e4117abbfea2879ddcd54]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:22.995378 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b5b0b1e46d14d78806859541e0ab3b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.995535 30916 tablet_service.cc:1467] Processing CreateTablet for tablet b605254185d140d8ab90fcaa5229f168 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b218cf6d6a9e4117abbfea2879ddcd54]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:22.995770 27741 tablet_service.cc:1467] Processing CreateTablet for tablet 9b5b0b1e46d14d78806859541e0ab3b6 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b218cf6d6a9e4117abbfea2879ddcd54]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:22.995903 30916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b605254185d140d8ab90fcaa5229f168. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.996287 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b5b0b1e46d14d78806859541e0ab3b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.998845 32187 tablet_bootstrap.cc:492] T 9b5b0b1e46d14d78806859541e0ab3b6 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:22.999336 30652 tablet_service.cc:1467] Processing CreateTablet for tablet b605254185d140d8ab90fcaa5229f168 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b218cf6d6a9e4117abbfea2879ddcd54]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:22.999792 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b605254185d140d8ab90fcaa5229f168. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:23.001055 32187 tablet_bootstrap.cc:654] T 9b5b0b1e46d14d78806859541e0ab3b6 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:23.007987 32186 tablet_bootstrap.cc:492] T b605254185d140d8ab90fcaa5229f168 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:23.010108 32196 tablet_bootstrap.cc:492] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:23.010208 32186 tablet_bootstrap.cc:654] T b605254185d140d8ab90fcaa5229f168 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:23.012352 32196 tablet_bootstrap.cc:654] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:22.989001 30784 tablet_service.cc:1467] Processing CreateTablet for tablet 87804550560d45269e6d4d6fe8938f55 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b218cf6d6a9e4117abbfea2879ddcd54]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:23.020051 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87804550560d45269e6d4d6fe8938f55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:23.021859 32196 tablet_bootstrap.cc:492] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:23.022156 32196 ts_tablet_manager.cc:1367] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20240501 08:12:23.023372 32196 raft_consensus.cc:348] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:23.023733 32196 raft_consensus.cc:374] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:23.023936 32196 raft_consensus.cc:729] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.024282 32196 consensus_queue.cc:260] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:23.025264 32196 ts_tablet_manager.cc:1398] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:23.028236 32196 tablet_bootstrap.cc:492] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:23.031405 32196 tablet_bootstrap.cc:654] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:23.039459 32200 tablet_bootstrap.cc:492] T 9b5b0b1e46d14d78806859541e0ab3b6 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:23.043416 32199 tablet_bootstrap.cc:492] T b605254185d140d8ab90fcaa5229f168 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:23.044569 32186 tablet_bootstrap.cc:492] T b605254185d140d8ab90fcaa5229f168 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:23.044945 32186 ts_tablet_manager.cc:1367] T b605254185d140d8ab90fcaa5229f168 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.008s
I20240501 08:12:23.046272 32186 raft_consensus.cc:348] T b605254185d140d8ab90fcaa5229f168 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:23.049901 32186 raft_consensus.cc:374] T b605254185d140d8ab90fcaa5229f168 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:23.050185 32186 raft_consensus.cc:729] T b605254185d140d8ab90fcaa5229f168 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.050573 32186 consensus_queue.cc:260] T b605254185d140d8ab90fcaa5229f168 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:23.051010 32187 tablet_bootstrap.cc:492] T 9b5b0b1e46d14d78806859541e0ab3b6 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:23.051381 32187 ts_tablet_manager.cc:1367] T 9b5b0b1e46d14d78806859541e0ab3b6 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.053s user 0.009s sys 0.023s
I20240501 08:12:23.052708 32186 ts_tablet_manager.cc:1398] T b605254185d140d8ab90fcaa5229f168 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 08:12:23.053258 32200 tablet_bootstrap.cc:654] T 9b5b0b1e46d14d78806859541e0ab3b6 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:23.055223 32199 tablet_bootstrap.cc:654] T b605254185d140d8ab90fcaa5229f168 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:23.056171 32202 tablet_bootstrap.cc:492] T 87804550560d45269e6d4d6fe8938f55 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:23.060185 32187 raft_consensus.cc:348] T 9b5b0b1e46d14d78806859541e0ab3b6 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:23.060608 32187 raft_consensus.cc:374] T 9b5b0b1e46d14d78806859541e0ab3b6 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:23.060796 32187 raft_consensus.cc:729] T 9b5b0b1e46d14d78806859541e0ab3b6 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.061199 32187 consensus_queue.cc:260] T 9b5b0b1e46d14d78806859541e0ab3b6 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:23.062140 32196 tablet_bootstrap.cc:492] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:23.062474 32196 ts_tablet_manager.cc:1367] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.004s
I20240501 08:12:23.063478 32187 ts_tablet_manager.cc:1398] T 9b5b0b1e46d14d78806859541e0ab3b6 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.012s user 0.001s sys 0.002s
I20240501 08:12:23.063987 32196 raft_consensus.cc:348] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:23.064378 32203 tablet_bootstrap.cc:492] T 9b5b0b1e46d14d78806859541e0ab3b6 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:23.064463 32196 raft_consensus.cc:374] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:23.064814 32196 raft_consensus.cc:729] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.065851 32196 consensus_queue.cc:260] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:23.068110 32200 tablet_bootstrap.cc:492] T 9b5b0b1e46d14d78806859541e0ab3b6 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:12:23.068186 32199 tablet_bootstrap.cc:492] T b605254185d140d8ab90fcaa5229f168 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:12:23.068367 32200 ts_tablet_manager.cc:1367] T 9b5b0b1e46d14d78806859541e0ab3b6 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.002s
I20240501 08:12:23.068564 32199 ts_tablet_manager.cc:1367] T b605254185d140d8ab90fcaa5229f168 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.011s
I20240501 08:12:23.069537 32199 raft_consensus.cc:348] T b605254185d140d8ab90fcaa5229f168 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:23.069600 32200 raft_consensus.cc:348] T 9b5b0b1e46d14d78806859541e0ab3b6 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:23.069831 32200 raft_consensus.cc:374] T 9b5b0b1e46d14d78806859541e0ab3b6 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:23.069890 32199 raft_consensus.cc:374] T b605254185d140d8ab90fcaa5229f168 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:23.070003 32200 raft_consensus.cc:729] T 9b5b0b1e46d14d78806859541e0ab3b6 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.070075 32203 tablet_bootstrap.cc:654] T 9b5b0b1e46d14d78806859541e0ab3b6 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:23.070199 32199 raft_consensus.cc:729] T b605254185d140d8ab90fcaa5229f168 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.070390 32200 consensus_queue.cc:260] T 9b5b0b1e46d14d78806859541e0ab3b6 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:23.071110 32200 ts_tablet_manager.cc:1398] T 9b5b0b1e46d14d78806859541e0ab3b6 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:12:23.072362 32202 tablet_bootstrap.cc:654] T 87804550560d45269e6d4d6fe8938f55 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:23.070655 32199 consensus_queue.cc:260] T b605254185d140d8ab90fcaa5229f168 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:23.073073 32196 ts_tablet_manager.cc:1398] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.010s user 0.000s sys 0.003s
I20240501 08:12:23.075002 32199 ts_tablet_manager.cc:1398] T b605254185d140d8ab90fcaa5229f168 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240501 08:12:23.078585 31980 raft_consensus.cc:482] T 9b5b0b1e46d14d78806859541e0ab3b6 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:23.078765 31980 raft_consensus.cc:504] T 9b5b0b1e46d14d78806859541e0ab3b6 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:23.079896 31980 leader_election.cc:288] T 9b5b0b1e46d14d78806859541e0ab3b6 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:23.081091 32187 tablet_bootstrap.cc:492] T 87804550560d45269e6d4d6fe8938f55 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:23.081101 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b5b0b1e46d14d78806859541e0ab3b6" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:23.082742 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b5b0b1e46d14d78806859541e0ab3b6" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:23.083024 30803 raft_consensus.cc:2454] T 9b5b0b1e46d14d78806859541e0ab3b6 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:12:23.083264 32187 tablet_bootstrap.cc:654] T 87804550560d45269e6d4d6fe8938f55 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:23.084241 27695 leader_election.cc:302] T 9b5b0b1e46d14d78806859541e0ab3b6 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 64998323a1f4482d8704bf20594d13a5; no voters:
I20240501 08:12:23.085711 31980 raft_consensus.cc:2781] T 9b5b0b1e46d14d78806859541e0ab3b6 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240501 08:12:23.086328 27697 leader_election.cc:341] T 9b5b0b1e46d14d78806859541e0ab3b6 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:23.087304 31980 raft_consensus.cc:482] T 9b5b0b1e46d14d78806859541e0ab3b6 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:23.090082 31980 raft_consensus.cc:3037] T 9b5b0b1e46d14d78806859541e0ab3b6 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:23.096361 32203 tablet_bootstrap.cc:492] T 9b5b0b1e46d14d78806859541e0ab3b6 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:23.096710 32202 tablet_bootstrap.cc:492] T 87804550560d45269e6d4d6fe8938f55 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:12:23.096948 32202 ts_tablet_manager.cc:1367] T 87804550560d45269e6d4d6fe8938f55 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.004s
I20240501 08:12:23.096971 32203 ts_tablet_manager.cc:1367] T 9b5b0b1e46d14d78806859541e0ab3b6 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.033s user 0.000s sys 0.006s
I20240501 08:12:23.098125 32203 raft_consensus.cc:348] T 9b5b0b1e46d14d78806859541e0ab3b6 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:23.098270 32202 raft_consensus.cc:348] T 87804550560d45269e6d4d6fe8938f55 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:23.098611 32202 raft_consensus.cc:374] T 87804550560d45269e6d4d6fe8938f55 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:23.098831 32202 raft_consensus.cc:729] T 87804550560d45269e6d4d6fe8938f55 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.099337 32202 consensus_queue.cc:260] T 87804550560d45269e6d4d6fe8938f55 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:23.099824 32203 raft_consensus.cc:374] T 9b5b0b1e46d14d78806859541e0ab3b6 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:23.100221 32203 raft_consensus.cc:729] T 9b5b0b1e46d14d78806859541e0ab3b6 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.100279 32202 ts_tablet_manager.cc:1398] T 87804550560d45269e6d4d6fe8938f55 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240501 08:12:23.100970 32203 consensus_queue.cc:260] T 9b5b0b1e46d14d78806859541e0ab3b6 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:23.105980 32203 ts_tablet_manager.cc:1398] T 9b5b0b1e46d14d78806859541e0ab3b6 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 08:12:23.110500 31980 raft_consensus.cc:504] T 9b5b0b1e46d14d78806859541e0ab3b6 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:23.111342 31980 leader_election.cc:288] T 9b5b0b1e46d14d78806859541e0ab3b6 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Requested vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:23.112530 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b5b0b1e46d14d78806859541e0ab3b6" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:23.112820 30537 raft_consensus.cc:3037] T 9b5b0b1e46d14d78806859541e0ab3b6 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:23.116072 30537 raft_consensus.cc:2454] T 9b5b0b1e46d14d78806859541e0ab3b6 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:12:23.116451 32187 tablet_bootstrap.cc:492] T 87804550560d45269e6d4d6fe8938f55 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:23.116886 27697 leader_election.cc:302] T 9b5b0b1e46d14d78806859541e0ab3b6 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 6331f77dae9d452184e8cd0cbf5f1b43; no voters:
I20240501 08:12:23.116892 32187 ts_tablet_manager.cc:1367] T 87804550560d45269e6d4d6fe8938f55 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.011s
I20240501 08:12:23.118197 31980 raft_consensus.cc:2781] T 9b5b0b1e46d14d78806859541e0ab3b6 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:23.118403 32187 raft_consensus.cc:348] T 87804550560d45269e6d4d6fe8938f55 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:23.118793 32187 raft_consensus.cc:374] T 87804550560d45269e6d4d6fe8938f55 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:23.119021 32187 raft_consensus.cc:729] T 87804550560d45269e6d4d6fe8938f55 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.119448 32187 consensus_queue.cc:260] T 87804550560d45269e6d4d6fe8938f55 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:23.118517 31980 raft_consensus.cc:686] T 9b5b0b1e46d14d78806859541e0ab3b6 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:12:23.123770 31980 consensus_queue.cc:237] T 9b5b0b1e46d14d78806859541e0ab3b6 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:23.125032 32185 raft_consensus.cc:482] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:23.125300 32185 raft_consensus.cc:504] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:23.126571 32185 leader_election.cc:288] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:12:23.127399 32187 ts_tablet_manager.cc:1398] T 87804550560d45269e6d4d6fe8938f55 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.010s user 0.000s sys 0.009s
I20240501 08:12:23.130420 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87804550560d45269e6d4d6fe8938f55" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:23.130739 30803 raft_consensus.cc:2454] T 87804550560d45269e6d4d6fe8938f55 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:23.131475 30871 leader_election.cc:302] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:23.133749 32185 raft_consensus.cc:2781] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:23.133944 32185 raft_consensus.cc:482] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:23.134093 32185 raft_consensus.cc:3037] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:23.137732 27620 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:23.143945 32185 raft_consensus.cc:504] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:23.143899 27618 catalog_manager.cc:5484] T 9b5b0b1e46d14d78806859541e0ab3b6 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:23.146831 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87804550560d45269e6d4d6fe8938f55" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:23.147092 30803 raft_consensus.cc:3037] T 87804550560d45269e6d4d6fe8938f55 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:23.150032 28021 raft_consensus.cc:1261] T 42662587e09e4c98b1830e4f650a4079 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:23.150684 30803 raft_consensus.cc:2454] T 87804550560d45269e6d4d6fe8938f55 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:23.150672 32112 consensus_queue.cc:1035] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:23.151247 30871 leader_election.cc:302] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:23.151683 32013 raft_consensus.cc:2781] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:23.152297 30803 raft_consensus.cc:1261] T 42662587e09e4c98b1830e4f650a4079 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:23.152974 32112 consensus_queue.cc:1035] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:23.161155 32013 raft_consensus.cc:686] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Becoming Leader. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:12:23.162077 32013 consensus_queue.cc:237] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:23.179677 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87804550560d45269e6d4d6fe8938f55" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:12:23.179986 27890 raft_consensus.cc:2454] T 87804550560d45269e6d4d6fe8938f55 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:23.196159 32185 leader_election.cc:288] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Requested vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:12:23.196892 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87804550560d45269e6d4d6fe8938f55" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:12:23.197194 27890 raft_consensus.cc:3037] T 87804550560d45269e6d4d6fe8938f55 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:23.204038 27890 raft_consensus.cc:2454] T 87804550560d45269e6d4d6fe8938f55 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:23.228068 27613 catalog_manager.cc:5576] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503) does not have the latest schema for tablet 42662587e09e4c98b1830e4f650a4079 (table rebalancer_dynamic_table [id=e04c4dccf30b4007a005b103ee9e5531]). Expected version 1 got 0
I20240501 08:12:23.230491 27618 catalog_manager.cc:5484] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: term changed from 0 to 1, leader changed from <none> to a29cd590999b4f32849ff5689cd53c11 (127.21.84.136). New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:23.235944 32223 tablet.cc:1722] T 42662587e09e4c98b1830e4f650a4079 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:23.238476 27613 catalog_manager.cc:5576] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503) does not have the latest schema for tablet 42662587e09e4c98b1830e4f650a4079 (table rebalancer_dynamic_table [id=e04c4dccf30b4007a005b103ee9e5531]). Expected version 1 got 0
W20240501 08:12:23.239065 30870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.134:39933, user_credentials={real_user=slave}} blocked reactor thread for 71875.9us
W20240501 08:12:23.240716 30737 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.132:41323, user_credentials={real_user=slave}} blocked reactor thread for 130058us
I20240501 08:12:23.233188 32222 tablet.cc:1722] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:23.245594 27613 catalog_manager.cc:5576] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933) does not have the latest schema for tablet 42662587e09e4c98b1830e4f650a4079 (table rebalancer_dynamic_table [id=e04c4dccf30b4007a005b103ee9e5531]). Expected version 1 got 0
I20240501 08:12:23.249971 27613 catalog_manager.cc:5576] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305) does not have the latest schema for tablet 42662587e09e4c98b1830e4f650a4079 (table rebalancer_dynamic_table [id=e04c4dccf30b4007a005b103ee9e5531]). Expected version 1 got 0
I20240501 08:12:23.250797 32225 tablet.cc:1722] T 42662587e09e4c98b1830e4f650a4079 P aad81047009a4ffcbc450b60030583b5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:23.251730 32225 tablet.cc:1620] T 42662587e09e4c98b1830e4f650a4079 P aad81047009a4ffcbc450b60030583b5: MemRowSet was empty: no flush needed.
I20240501 08:12:23.255074 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b5b0b1e46d14d78806859541e0ab3b6" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:23.255337 30803 raft_consensus.cc:3037] T 9b5b0b1e46d14d78806859541e0ab3b6 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:23.260026 30803 raft_consensus.cc:2454] T 9b5b0b1e46d14d78806859541e0ab3b6 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:12:23.259037 27613 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=e04c4dccf30b4007a005b103ee9e5531] alter complete (version 1)
I20240501 08:12:23.242003 32222 tablet.cc:1620] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:23.236748 32223 tablet.cc:1620] T 42662587e09e4c98b1830e4f650a4079 P 64998323a1f4482d8704bf20594d13a5: MemRowSet was empty: no flush needed.
I20240501 08:12:23.380900 27620 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:23.390607 32223 tablet.cc:1722] T 42662587e09e4c98b1830e4f650a4079 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:23.391155 32223 tablet.cc:1620] T 42662587e09e4c98b1830e4f650a4079 P 64998323a1f4482d8704bf20594d13a5: MemRowSet was empty: no flush needed.
I20240501 08:12:23.391958 32222 tablet.cc:1722] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:23.392596 32222 tablet.cc:1620] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:23.393817 32225 tablet.cc:1722] T 42662587e09e4c98b1830e4f650a4079 P aad81047009a4ffcbc450b60030583b5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:23.394479 32225 tablet.cc:1620] T 42662587e09e4c98b1830e4f650a4079 P aad81047009a4ffcbc450b60030583b5: MemRowSet was empty: no flush needed.
I20240501 08:12:23.408421 27620 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=e04c4dccf30b4007a005b103ee9e5531] alter complete (version 2)
I20240501 08:12:23.423300 32220 raft_consensus.cc:482] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:23.423629 32220 raft_consensus.cc:504] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:23.424713 32220 leader_election.cc:288] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:23.425227 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b605254185d140d8ab90fcaa5229f168" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:12:23.425487 28151 raft_consensus.cc:2454] T b605254185d140d8ab90fcaa5229f168 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:23.425599 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b605254185d140d8ab90fcaa5229f168" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:23.425863 30672 raft_consensus.cc:2454] T b605254185d140d8ab90fcaa5229f168 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:23.426014 30870 leader_election.cc:302] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:23.426393 32220 raft_consensus.cc:2781] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:23.426558 32220 raft_consensus.cc:482] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:23.426687 32220 raft_consensus.cc:3037] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:23.429342 32220 raft_consensus.cc:504] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:23.429993 32220 leader_election.cc:288] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Requested vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:23.430557 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b605254185d140d8ab90fcaa5229f168" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:23.430842 30672 raft_consensus.cc:3037] T b605254185d140d8ab90fcaa5229f168 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:23.430842 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b605254185d140d8ab90fcaa5229f168" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:12:23.431131 28151 raft_consensus.cc:3037] T b605254185d140d8ab90fcaa5229f168 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:23.434417 30672 raft_consensus.cc:2454] T b605254185d140d8ab90fcaa5229f168 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:23.434463 28151 raft_consensus.cc:2454] T b605254185d140d8ab90fcaa5229f168 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:23.434927 30870 leader_election.cc:302] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a022e1a501f4d44a0468054d6c3de47, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:23.435263 32220 raft_consensus.cc:2781] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:23.435475 32220 raft_consensus.cc:686] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Becoming Leader. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:12:23.435950 32220 consensus_queue.cc:237] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:23.440956 27620 catalog_manager.cc:5484] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: term changed from 0 to 1, leader changed from <none> to a29cd590999b4f32849ff5689cd53c11 (127.21.84.136). New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:23.509408 27620 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:23.517241 32222 tablet.cc:1722] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:23.517693 32223 tablet.cc:1722] T 42662587e09e4c98b1830e4f650a4079 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:23.526293 32225 tablet.cc:1722] T 42662587e09e4c98b1830e4f650a4079 P aad81047009a4ffcbc450b60030583b5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:23.539186 27620 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table_new_name [id=e04c4dccf30b4007a005b103ee9e5531] alter complete (version 3)
I20240501 08:12:23.557098 32214 consensus_queue.cc:1035] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:23.564949 32214 consensus_queue.cc:1035] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:23.583469 32118 consensus_queue.cc:1035] T 9b5b0b1e46d14d78806859541e0ab3b6 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:23.595686 31980 consensus_queue.cc:1035] T 9b5b0b1e46d14d78806859541e0ab3b6 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:23.642823 27613 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:23.643348 27613 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:23.646369 27613 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 42662587e09e4c98b1830e4f650a4079
I20240501 08:12:23.647320 30652 tablet_service.cc:1514] Processing DeleteTablet for tablet 42662587e09e4c98b1830e4f650a4079 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:23 UTC) from {username='slave'} at 127.0.0.1:57156
I20240501 08:12:23.647316 30784 tablet_service.cc:1514] Processing DeleteTablet for tablet 42662587e09e4c98b1830e4f650a4079 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:23 UTC) from {username='slave'} at 127.0.0.1:38952
I20240501 08:12:23.647372 28002 tablet_service.cc:1514] Processing DeleteTablet for tablet 42662587e09e4c98b1830e4f650a4079 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:23 UTC) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:23.648348 32245 tablet_replica.cc:330] T 42662587e09e4c98b1830e4f650a4079 P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:23.648581 32245 raft_consensus.cc:2229] T 42662587e09e4c98b1830e4f650a4079 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:23.648844 32245 raft_consensus.cc:2258] T 42662587e09e4c98b1830e4f650a4079 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:23.649737 32245 ts_tablet_manager.cc:1875] T 42662587e09e4c98b1830e4f650a4079 P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:23.656813 32245 ts_tablet_manager.cc:1888] T 42662587e09e4c98b1830e4f650a4079 P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:23.656997 32245 log.cc:1196] T 42662587e09e4c98b1830e4f650a4079 P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/42662587e09e4c98b1830e4f650a4079
I20240501 08:12:23.657562 32245 ts_tablet_manager.cc:1909] T 42662587e09e4c98b1830e4f650a4079 P aad81047009a4ffcbc450b60030583b5: Deleting consensus metadata
I20240501 08:12:23.658564 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet 42662587e09e4c98b1830e4f650a4079 (table rebalancer_dynamic_table_new_name [id=e04c4dccf30b4007a005b103ee9e5531]) successfully deleted
I20240501 08:12:23.660353 32243 tablet_replica.cc:330] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47: stopping tablet replica
I20240501 08:12:23.660733 32243 raft_consensus.cc:2229] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:23.661317 32243 raft_consensus.cc:2258] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:23.662190 32244 tablet_replica.cc:330] T 42662587e09e4c98b1830e4f650a4079 P 64998323a1f4482d8704bf20594d13a5: stopping tablet replica
I20240501 08:12:23.662321 32243 ts_tablet_manager.cc:1875] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:23.662439 32244 raft_consensus.cc:2229] T 42662587e09e4c98b1830e4f650a4079 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:23.662704 32244 raft_consensus.cc:2258] T 42662587e09e4c98b1830e4f650a4079 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:23.664256 32244 ts_tablet_manager.cc:1875] T 42662587e09e4c98b1830e4f650a4079 P 64998323a1f4482d8704bf20594d13a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:23.669553 32243 ts_tablet_manager.cc:1888] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:23.669728 32243 log.cc:1196] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/wals/42662587e09e4c98b1830e4f650a4079
I20240501 08:12:23.670316 32243 ts_tablet_manager.cc:1909] T 42662587e09e4c98b1830e4f650a4079 P 9a022e1a501f4d44a0468054d6c3de47: Deleting consensus metadata
I20240501 08:12:23.671258 27604 catalog_manager.cc:4830] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): tablet 42662587e09e4c98b1830e4f650a4079 (table rebalancer_dynamic_table_new_name [id=e04c4dccf30b4007a005b103ee9e5531]) successfully deleted
I20240501 08:12:23.671550 32244 ts_tablet_manager.cc:1888] T 42662587e09e4c98b1830e4f650a4079 P 64998323a1f4482d8704bf20594d13a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:23.671682 32244 log.cc:1196] T 42662587e09e4c98b1830e4f650a4079 P 64998323a1f4482d8704bf20594d13a5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal/wals/42662587e09e4c98b1830e4f650a4079
I20240501 08:12:23.672106 32244 ts_tablet_manager.cc:1909] T 42662587e09e4c98b1830e4f650a4079 P 64998323a1f4482d8704bf20594d13a5: Deleting consensus metadata
I20240501 08:12:23.672928 27605 catalog_manager.cc:4830] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): tablet 42662587e09e4c98b1830e4f650a4079 (table rebalancer_dynamic_table_new_name [id=e04c4dccf30b4007a005b103ee9e5531]) successfully deleted
I20240501 08:12:23.754004 27613 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:23.762285 30915 tablet_service.cc:1467] Processing CreateTablet for tablet 759b87b7dafb486b802410fab18bfb11 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d4c3fec0dd48477c9d16bc0f521c41e3]), partition=
I20240501 08:12:23.762800 30915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 759b87b7dafb486b802410fab18bfb11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:23.762985 30517 tablet_service.cc:1467] Processing CreateTablet for tablet 759b87b7dafb486b802410fab18bfb11 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d4c3fec0dd48477c9d16bc0f521c41e3]), partition=
I20240501 08:12:23.763029 30784 tablet_service.cc:1467] Processing CreateTablet for tablet 759b87b7dafb486b802410fab18bfb11 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d4c3fec0dd48477c9d16bc0f521c41e3]), partition=
I20240501 08:12:23.763424 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 759b87b7dafb486b802410fab18bfb11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:23.764098 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 759b87b7dafb486b802410fab18bfb11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:23.775660 32246 tablet_bootstrap.cc:492] T 759b87b7dafb486b802410fab18bfb11 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:23.775789 32247 tablet_bootstrap.cc:492] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:23.778002 32248 tablet_bootstrap.cc:492] T 759b87b7dafb486b802410fab18bfb11 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:23.779209 32246 tablet_bootstrap.cc:654] T 759b87b7dafb486b802410fab18bfb11 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:23.779579 32248 tablet_bootstrap.cc:654] T 759b87b7dafb486b802410fab18bfb11 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:23.779753 32247 tablet_bootstrap.cc:654] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:23.781435 32246 tablet_bootstrap.cc:492] T 759b87b7dafb486b802410fab18bfb11 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:23.781581 32246 ts_tablet_manager.cc:1367] T 759b87b7dafb486b802410fab18bfb11 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 08:12:23.782229 32246 raft_consensus.cc:348] T 759b87b7dafb486b802410fab18bfb11 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:23.782413 32246 raft_consensus.cc:374] T 759b87b7dafb486b802410fab18bfb11 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:23.782518 32246 raft_consensus.cc:729] T 759b87b7dafb486b802410fab18bfb11 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.782759 32246 consensus_queue.cc:260] T 759b87b7dafb486b802410fab18bfb11 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:23.783357 32246 ts_tablet_manager.cc:1398] T 759b87b7dafb486b802410fab18bfb11 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:23.785439 32247 tablet_bootstrap.cc:492] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:23.785647 32247 ts_tablet_manager.cc:1367] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20240501 08:12:23.786859 32247 raft_consensus.cc:348] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:23.787211 32247 raft_consensus.cc:374] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:23.787335 32247 raft_consensus.cc:729] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.787695 32247 consensus_queue.cc:260] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:23.788481 32247 ts_tablet_manager.cc:1398] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240501 08:12:23.790639 32248 tablet_bootstrap.cc:492] T 759b87b7dafb486b802410fab18bfb11 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:23.790817 32248 ts_tablet_manager.cc:1367] T 759b87b7dafb486b802410fab18bfb11 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20240501 08:12:23.792078 32248 raft_consensus.cc:348] T 759b87b7dafb486b802410fab18bfb11 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:23.792362 32248 raft_consensus.cc:374] T 759b87b7dafb486b802410fab18bfb11 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:23.792516 32248 raft_consensus.cc:729] T 759b87b7dafb486b802410fab18bfb11 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.792775 32248 consensus_queue.cc:260] T 759b87b7dafb486b802410fab18bfb11 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:23.794930 32248 ts_tablet_manager.cc:1398] T 759b87b7dafb486b802410fab18bfb11 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:23.808805 32227 raft_consensus.cc:482] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:23.809096 32227 raft_consensus.cc:504] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:23.810115 32227 leader_election.cc:288] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:23.810585 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "759b87b7dafb486b802410fab18bfb11" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:23.810674 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "759b87b7dafb486b802410fab18bfb11" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:23.810928 30803 raft_consensus.cc:2454] T 759b87b7dafb486b802410fab18bfb11 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 0.
I20240501 08:12:23.810956 30936 raft_consensus.cc:2454] T 759b87b7dafb486b802410fab18bfb11 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 0.
I20240501 08:12:23.811470 30472 leader_election.cc:302] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:23.811774 32227 raft_consensus.cc:2781] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:23.811939 32227 raft_consensus.cc:482] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:23.812098 32227 raft_consensus.cc:3037] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:23.814713 32227 raft_consensus.cc:504] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:23.815412 32227 leader_election.cc:288] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 election: Requested vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:23.816211 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "759b87b7dafb486b802410fab18bfb11" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:23.816247 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "759b87b7dafb486b802410fab18bfb11" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:23.816462 30936 raft_consensus.cc:3037] T 759b87b7dafb486b802410fab18bfb11 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:23.816514 30804 raft_consensus.cc:3037] T 759b87b7dafb486b802410fab18bfb11 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:23.819934 30804 raft_consensus.cc:2454] T 759b87b7dafb486b802410fab18bfb11 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 1.
I20240501 08:12:23.820086 30936 raft_consensus.cc:2454] T 759b87b7dafb486b802410fab18bfb11 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 1.
I20240501 08:12:23.820617 30471 leader_election.cc:302] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 64998323a1f4482d8704bf20594d13a5; no voters:
I20240501 08:12:23.821005 32227 raft_consensus.cc:2781] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:23.821607 32227 raft_consensus.cc:686] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Becoming Leader. State: Replica: 31811708c6774037b82df44024b229b6, State: Running, Role: LEADER
I20240501 08:12:23.821909 32227 consensus_queue.cc:237] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:23.825660 27613 catalog_manager.cc:5484] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31811708c6774037b82df44024b229b6 (127.21.84.133). New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:23.939404 27613 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:23.945991 30936 raft_consensus.cc:1261] T 759b87b7dafb486b802410fab18bfb11 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:23.946892 32228 consensus_queue.cc:1035] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:23.947508 30804 raft_consensus.cc:1261] T 759b87b7dafb486b802410fab18bfb11 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:23.948140 32228 consensus_queue.cc:1035] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:23.952419 32255 tablet.cc:1722] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:23.953164 32255 tablet.cc:1620] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:23.954056 27620 catalog_manager.cc:5576] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503) does not have the latest schema for tablet 759b87b7dafb486b802410fab18bfb11 (table rebalancer_dynamic_table [id=d4c3fec0dd48477c9d16bc0f521c41e3]). Expected version 1 got 0
I20240501 08:12:23.955302 32223 tablet.cc:1722] T 759b87b7dafb486b802410fab18bfb11 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:23.955919 32223 tablet.cc:1620] T 759b87b7dafb486b802410fab18bfb11 P 64998323a1f4482d8704bf20594d13a5: MemRowSet was empty: no flush needed.
I20240501 08:12:23.958300 32256 tablet.cc:1722] T 759b87b7dafb486b802410fab18bfb11 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:23.959550 32256 tablet.cc:1620] T 759b87b7dafb486b802410fab18bfb11 P a29cd590999b4f32849ff5689cd53c11: MemRowSet was empty: no flush needed.
I20240501 08:12:23.963284 27617 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_extra_table_42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:23.966673 27620 catalog_manager.cc:5576] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845) does not have the latest schema for tablet 759b87b7dafb486b802410fab18bfb11 (table rebalancer_dynamic_table [id=d4c3fec0dd48477c9d16bc0f521c41e3]). Expected version 1 got 0
I20240501 08:12:23.967892 27618 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=d4c3fec0dd48477c9d16bc0f521c41e3] alter complete (version 1)
I20240501 08:12:23.976154 30784 tablet_service.cc:1467] Processing CreateTablet for tablet 325b3dbba80d491dab81840e8eb171e9 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=6aa96b40a8e74b81bfc5ccab950a210d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:23.976457 28132 tablet_service.cc:1467] Processing CreateTablet for tablet 325b3dbba80d491dab81840e8eb171e9 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=6aa96b40a8e74b81bfc5ccab950a210d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:23.976612 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 325b3dbba80d491dab81840e8eb171e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:23.976987 28132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 325b3dbba80d491dab81840e8eb171e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:23.977869 28002 tablet_service.cc:1467] Processing CreateTablet for tablet 87bd346903dc4c2c85ecaf362fe57799 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=6aa96b40a8e74b81bfc5ccab950a210d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:23.978389 28002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87bd346903dc4c2c85ecaf362fe57799. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:23.983160 28131 tablet_service.cc:1467] Processing CreateTablet for tablet d3673fe03fb64c47b30dc2b79042f44a (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=6aa96b40a8e74b81bfc5ccab950a210d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:23.983163 30782 tablet_service.cc:1467] Processing CreateTablet for tablet d3673fe03fb64c47b30dc2b79042f44a (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=6aa96b40a8e74b81bfc5ccab950a210d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:23.983608 30782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3673fe03fb64c47b30dc2b79042f44a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:23.983611 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3673fe03fb64c47b30dc2b79042f44a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:23.983508 30652 tablet_service.cc:1467] Processing CreateTablet for tablet 325b3dbba80d491dab81840e8eb171e9 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=6aa96b40a8e74b81bfc5ccab950a210d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:23.983970 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 325b3dbba80d491dab81840e8eb171e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:23.986322 30514 tablet_service.cc:1467] Processing CreateTablet for tablet d3673fe03fb64c47b30dc2b79042f44a (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=6aa96b40a8e74b81bfc5ccab950a210d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:23.986750 30514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3673fe03fb64c47b30dc2b79042f44a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:23.990550 30517 tablet_service.cc:1467] Processing CreateTablet for tablet 87bd346903dc4c2c85ecaf362fe57799 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=6aa96b40a8e74b81bfc5ccab950a210d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:23.991015 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87bd346903dc4c2c85ecaf362fe57799. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:23.994597 30650 tablet_service.cc:1467] Processing CreateTablet for tablet 87bd346903dc4c2c85ecaf362fe57799 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=6aa96b40a8e74b81bfc5ccab950a210d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:23.995116 30650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87bd346903dc4c2c85ecaf362fe57799. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:23.995888 32220 consensus_queue.cc:1035] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:24.018118 32247 tablet_bootstrap.cc:492] T d3673fe03fb64c47b30dc2b79042f44a P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:24.018862 32248 tablet_bootstrap.cc:492] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:24.019786 32259 tablet_bootstrap.cc:492] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:24.020243 32247 tablet_bootstrap.cc:654] T d3673fe03fb64c47b30dc2b79042f44a P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:24.025820 32259 tablet_bootstrap.cc:654] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:24.027648 32248 tablet_bootstrap.cc:654] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:24.030979 32261 tablet_bootstrap.cc:492] T 325b3dbba80d491dab81840e8eb171e9 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:24.034076 32220 consensus_queue.cc:1035] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:24.037654 32259 tablet_bootstrap.cc:492] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:12:24.038432 32259 ts_tablet_manager.cc:1367] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.002s
I20240501 08:12:24.040417 32259 raft_consensus.cc:348] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:24.042305 32259 raft_consensus.cc:374] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:24.042582 32259 raft_consensus.cc:729] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:12:24.043179 32259 consensus_queue.cc:260] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:24.044688 32248 tablet_bootstrap.cc:492] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:24.044978 32248 ts_tablet_manager.cc:1367] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.008s
I20240501 08:12:24.047170 32259 ts_tablet_manager.cc:1398] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240501 08:12:24.049541 32248 raft_consensus.cc:348] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:24.049835 32248 raft_consensus.cc:374] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:24.049966 32248 raft_consensus.cc:729] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:24.050781 32261 tablet_bootstrap.cc:654] T 325b3dbba80d491dab81840e8eb171e9 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:24.053246 32247 tablet_bootstrap.cc:492] T d3673fe03fb64c47b30dc2b79042f44a P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:24.053474 32247 ts_tablet_manager.cc:1367] T d3673fe03fb64c47b30dc2b79042f44a P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.006s
I20240501 08:12:24.053521 32248 consensus_queue.cc:260] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:24.054353 32247 raft_consensus.cc:348] T d3673fe03fb64c47b30dc2b79042f44a P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:24.054646 32247 raft_consensus.cc:374] T d3673fe03fb64c47b30dc2b79042f44a P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:24.054740 32247 raft_consensus.cc:729] T d3673fe03fb64c47b30dc2b79042f44a P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:24.055030 32247 consensus_queue.cc:260] T d3673fe03fb64c47b30dc2b79042f44a P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:24.056288 32248 ts_tablet_manager.cc:1398] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 08:12:24.056699 32248 tablet_bootstrap.cc:492] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:24.059065 32261 tablet_bootstrap.cc:492] T 325b3dbba80d491dab81840e8eb171e9 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:24.059652 32247 ts_tablet_manager.cc:1398] T d3673fe03fb64c47b30dc2b79042f44a P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240501 08:12:24.060278 32261 ts_tablet_manager.cc:1367] T 325b3dbba80d491dab81840e8eb171e9 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.003s
I20240501 08:12:24.061581 32248 tablet_bootstrap.cc:654] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:24.061878 32257 tablet_bootstrap.cc:492] T 325b3dbba80d491dab81840e8eb171e9 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:24.062222 32247 tablet_bootstrap.cc:492] T 87bd346903dc4c2c85ecaf362fe57799 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:24.063081 32261 raft_consensus.cc:348] T 325b3dbba80d491dab81840e8eb171e9 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:24.065353 32247 tablet_bootstrap.cc:654] T 87bd346903dc4c2c85ecaf362fe57799 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:24.069092 32261 raft_consensus.cc:374] T 325b3dbba80d491dab81840e8eb171e9 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:24.069299 32261 raft_consensus.cc:729] T 325b3dbba80d491dab81840e8eb171e9 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:24.069641 32261 consensus_queue.cc:260] T 325b3dbba80d491dab81840e8eb171e9 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:24.071151 32257 tablet_bootstrap.cc:654] T 325b3dbba80d491dab81840e8eb171e9 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:24.071549 32248 tablet_bootstrap.cc:492] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:24.071697 32248 ts_tablet_manager.cc:1367] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.001s
I20240501 08:12:24.073120 32248 raft_consensus.cc:348] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:24.073411 32248 raft_consensus.cc:374] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:24.073559 32248 raft_consensus.cc:729] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:24.073915 32248 consensus_queue.cc:260] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:24.075171 32261 ts_tablet_manager.cc:1398] T 325b3dbba80d491dab81840e8eb171e9 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240501 08:12:24.075362 32247 tablet_bootstrap.cc:492] T 87bd346903dc4c2c85ecaf362fe57799 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:24.075505 32247 ts_tablet_manager.cc:1367] T 87bd346903dc4c2c85ecaf362fe57799 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.009s
I20240501 08:12:24.076581 32261 tablet_bootstrap.cc:492] T 87bd346903dc4c2c85ecaf362fe57799 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:24.077728 27620 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:24.078238 32247 raft_consensus.cc:348] T 87bd346903dc4c2c85ecaf362fe57799 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:24.078513 32261 tablet_bootstrap.cc:654] T 87bd346903dc4c2c85ecaf362fe57799 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:24.078537 32247 raft_consensus.cc:374] T 87bd346903dc4c2c85ecaf362fe57799 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:24.078641 32247 raft_consensus.cc:729] T 87bd346903dc4c2c85ecaf362fe57799 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:24.078922 32247 consensus_queue.cc:260] T 87bd346903dc4c2c85ecaf362fe57799 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:24.079633 32247 ts_tablet_manager.cc:1398] T 87bd346903dc4c2c85ecaf362fe57799 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:24.080111 32248 ts_tablet_manager.cc:1398] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20240501 08:12:24.083995 32257 tablet_bootstrap.cc:492] T 325b3dbba80d491dab81840e8eb171e9 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:12:24.084254 32257 ts_tablet_manager.cc:1367] T 325b3dbba80d491dab81840e8eb171e9 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20240501 08:12:24.085773 32257 raft_consensus.cc:348] T 325b3dbba80d491dab81840e8eb171e9 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:24.086086 32257 raft_consensus.cc:374] T 325b3dbba80d491dab81840e8eb171e9 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:24.086246 32257 raft_consensus.cc:729] T 325b3dbba80d491dab81840e8eb171e9 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:12:24.086555 32257 consensus_queue.cc:260] T 325b3dbba80d491dab81840e8eb171e9 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:24.087858 32257 ts_tablet_manager.cc:1398] T 325b3dbba80d491dab81840e8eb171e9 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:24.088196 32257 tablet_bootstrap.cc:492] T d3673fe03fb64c47b30dc2b79042f44a P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:24.090356 32257 tablet_bootstrap.cc:654] T d3673fe03fb64c47b30dc2b79042f44a P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:24.090873 32255 tablet.cc:1722] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:24.091385 32255 tablet.cc:1620] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:24.091961 32223 tablet.cc:1722] T 759b87b7dafb486b802410fab18bfb11 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:24.092440 32223 tablet.cc:1620] T 759b87b7dafb486b802410fab18bfb11 P 64998323a1f4482d8704bf20594d13a5: MemRowSet was empty: no flush needed.
I20240501 08:12:24.097952 32256 tablet.cc:1722] T 759b87b7dafb486b802410fab18bfb11 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:24.098428 32256 tablet.cc:1620] T 759b87b7dafb486b802410fab18bfb11 P a29cd590999b4f32849ff5689cd53c11: MemRowSet was empty: no flush needed.
I20240501 08:12:24.100018 32261 tablet_bootstrap.cc:492] T 87bd346903dc4c2c85ecaf362fe57799 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:24.100286 32261 ts_tablet_manager.cc:1367] T 87bd346903dc4c2c85ecaf362fe57799 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.001s
I20240501 08:12:24.101264 32261 raft_consensus.cc:348] T 87bd346903dc4c2c85ecaf362fe57799 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:24.101656 32261 raft_consensus.cc:374] T 87bd346903dc4c2c85ecaf362fe57799 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:24.101785 32261 raft_consensus.cc:729] T 87bd346903dc4c2c85ecaf362fe57799 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:24.102144 32261 consensus_queue.cc:260] T 87bd346903dc4c2c85ecaf362fe57799 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:24.104746 32261 ts_tablet_manager.cc:1398] T 87bd346903dc4c2c85ecaf362fe57799 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:24.106628 32257 tablet_bootstrap.cc:492] T d3673fe03fb64c47b30dc2b79042f44a P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:12:24.106832 32257 ts_tablet_manager.cc:1367] T d3673fe03fb64c47b30dc2b79042f44a P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.003s
I20240501 08:12:24.107726 32257 raft_consensus.cc:348] T d3673fe03fb64c47b30dc2b79042f44a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:24.108002 32257 raft_consensus.cc:374] T d3673fe03fb64c47b30dc2b79042f44a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:24.108124 32257 raft_consensus.cc:729] T d3673fe03fb64c47b30dc2b79042f44a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:12:24.108415 32257 consensus_queue.cc:260] T d3673fe03fb64c47b30dc2b79042f44a P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:24.109043 32257 ts_tablet_manager.cc:1398] T d3673fe03fb64c47b30dc2b79042f44a P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:24.113154 27613 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=d4c3fec0dd48477c9d16bc0f521c41e3] alter complete (version 2)
I20240501 08:12:24.214109 32209 raft_consensus.cc:482] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:24.214320 32209 raft_consensus.cc:504] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:24.215121 32209 leader_election.cc:288] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:24.215245 27617 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:24.215552 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3673fe03fb64c47b30dc2b79042f44a" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:24.215829 30536 raft_consensus.cc:2454] T d3673fe03fb64c47b30dc2b79042f44a P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:24.216238 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3673fe03fb64c47b30dc2b79042f44a" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:12:24.216354 30740 leader_election.cc:302] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 64998323a1f4482d8704bf20594d13a5; no voters:
I20240501 08:12:24.216477 28151 raft_consensus.cc:2454] T d3673fe03fb64c47b30dc2b79042f44a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:24.216634 32209 raft_consensus.cc:2781] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:24.216739 32209 raft_consensus.cc:482] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:24.216822 32209 raft_consensus.cc:3037] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:24.220546 32209 raft_consensus.cc:504] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:24.221779 32209 leader_election.cc:288] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Requested vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:24.221786 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3673fe03fb64c47b30dc2b79042f44a" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:24.222055 30537 raft_consensus.cc:3037] T d3673fe03fb64c47b30dc2b79042f44a P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:24.222054 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3673fe03fb64c47b30dc2b79042f44a" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:12:24.222270 28151 raft_consensus.cc:3037] T d3673fe03fb64c47b30dc2b79042f44a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:24.224577 28151 raft_consensus.cc:2454] T d3673fe03fb64c47b30dc2b79042f44a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:24.225000 30737 leader_election.cc:302] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:12:24.225627 30537 raft_consensus.cc:2454] T d3673fe03fb64c47b30dc2b79042f44a P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:24.226450 32209 raft_consensus.cc:2781] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:24.226755 32209 raft_consensus.cc:686] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Becoming Leader. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Running, Role: LEADER
I20240501 08:12:24.227128 32209 consensus_queue.cc:237] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:24.230659 32255 tablet.cc:1722] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:24.231189 32223 tablet.cc:1722] T 759b87b7dafb486b802410fab18bfb11 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:24.233006 27613 catalog_manager.cc:5484] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64998323a1f4482d8704bf20594d13a5 (127.21.84.135). New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:24.240650 32227 raft_consensus.cc:482] T 87bd346903dc4c2c85ecaf362fe57799 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:24.240839 32227 raft_consensus.cc:504] T 87bd346903dc4c2c85ecaf362fe57799 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:24.241674 32227 leader_election.cc:288] T 87bd346903dc4c2c85ecaf362fe57799 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:24.243036 32218 raft_consensus.cc:482] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:24.243207 32218 raft_consensus.cc:504] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:24.243733 30671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87bd346903dc4c2c85ecaf362fe57799" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:24.243973 30671 raft_consensus.cc:2454] T 87bd346903dc4c2c85ecaf362fe57799 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 0.
I20240501 08:12:24.244141 32218 leader_election.cc:288] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:24.244455 30470 leader_election.cc:302] T 87bd346903dc4c2c85ecaf362fe57799 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:24.245577 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87bd346903dc4c2c85ecaf362fe57799" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:24.245868 30537 raft_consensus.cc:2454] T 87bd346903dc4c2c85ecaf362fe57799 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 0.
I20240501 08:12:24.246397 27958 leader_election.cc:302] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:12:24.246744 32218 raft_consensus.cc:2781] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:24.247234 30671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87bd346903dc4c2c85ecaf362fe57799" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:24.247470 30671 raft_consensus.cc:2454] T 87bd346903dc4c2c85ecaf362fe57799 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 0.
I20240501 08:12:24.247814 32218 raft_consensus.cc:482] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:24.248013 32218 raft_consensus.cc:3037] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:24.248539 32227 raft_consensus.cc:2781] T 87bd346903dc4c2c85ecaf362fe57799 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:24.248641 32227 raft_consensus.cc:482] T 87bd346903dc4c2c85ecaf362fe57799 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:24.248750 32227 raft_consensus.cc:3037] T 87bd346903dc4c2c85ecaf362fe57799 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:24.253273 27613 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table_new_name [id=d4c3fec0dd48477c9d16bc0f521c41e3] alter complete (version 3)
I20240501 08:12:24.255316 32218 raft_consensus.cc:504] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:24.256915 32218 leader_election.cc:288] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Requested vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:24.257226 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87bd346903dc4c2c85ecaf362fe57799" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:24.257622 30671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87bd346903dc4c2c85ecaf362fe57799" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:24.257804 30671 raft_consensus.cc:3037] T 87bd346903dc4c2c85ecaf362fe57799 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:24.260475 30671 raft_consensus.cc:2454] T 87bd346903dc4c2c85ecaf362fe57799 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 1.
I20240501 08:12:24.261534 27955 leader_election.cc:302] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a022e1a501f4d44a0468054d6c3de47, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:12:24.267180 32227 raft_consensus.cc:504] T 87bd346903dc4c2c85ecaf362fe57799 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:24.267544 30537 raft_consensus.cc:2379] T 87bd346903dc4c2c85ecaf362fe57799 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate aad81047009a4ffcbc450b60030583b5 in current term 1: Already voted for candidate 31811708c6774037b82df44024b229b6 in this term.
I20240501 08:12:24.268905 30671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87bd346903dc4c2c85ecaf362fe57799" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:24.269198 30671 raft_consensus.cc:2379] T 87bd346903dc4c2c85ecaf362fe57799 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 31811708c6774037b82df44024b229b6 in current term 1: Already voted for candidate aad81047009a4ffcbc450b60030583b5 in this term.
I20240501 08:12:24.269754 32227 leader_election.cc:288] T 87bd346903dc4c2c85ecaf362fe57799 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 election: Requested vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:24.271286 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87bd346903dc4c2c85ecaf362fe57799" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:12:24.271534 28021 raft_consensus.cc:2379] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31811708c6774037b82df44024b229b6 in current term 1: Already voted for candidate aad81047009a4ffcbc450b60030583b5 in this term.
I20240501 08:12:24.272212 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87bd346903dc4c2c85ecaf362fe57799" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:12:24.272434 28021 raft_consensus.cc:2379] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 31811708c6774037b82df44024b229b6 in current term 1: Already voted for candidate aad81047009a4ffcbc450b60030583b5 in this term.
I20240501 08:12:24.272742 32218 raft_consensus.cc:2781] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:24.272967 32256 tablet.cc:1722] T 759b87b7dafb486b802410fab18bfb11 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:24.273274 30472 leader_election.cc:302] T 87bd346903dc4c2c85ecaf362fe57799 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31811708c6774037b82df44024b229b6; no voters: 9a022e1a501f4d44a0468054d6c3de47, aad81047009a4ffcbc450b60030583b5
I20240501 08:12:24.273885 32228 raft_consensus.cc:2726] T 87bd346903dc4c2c85ecaf362fe57799 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 08:12:24.274154 32218 raft_consensus.cc:686] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Becoming Leader. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Running, Role: LEADER
I20240501 08:12:24.274430 32218 consensus_queue.cc:237] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:24.278651 27617 catalog_manager.cc:5484] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to aad81047009a4ffcbc450b60030583b5 (127.21.84.131). New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:24.346585 32209 raft_consensus.cc:482] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:24.346794 32209 raft_consensus.cc:504] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:24.347388 32209 leader_election.cc:288] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:24.347994 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "325b3dbba80d491dab81840e8eb171e9" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:12:24.348244 28151 raft_consensus.cc:2454] T 325b3dbba80d491dab81840e8eb171e9 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:24.348595 30737 leader_election.cc:302] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:12:24.348667 30669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "325b3dbba80d491dab81840e8eb171e9" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:24.348886 30669 raft_consensus.cc:2454] T 325b3dbba80d491dab81840e8eb171e9 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:24.348948 32209 raft_consensus.cc:2781] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:24.349045 32209 raft_consensus.cc:482] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:24.349155 32209 raft_consensus.cc:3037] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:24.351152 32209 raft_consensus.cc:504] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:24.351699 32209 leader_election.cc:288] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Requested vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:24.352134 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "325b3dbba80d491dab81840e8eb171e9" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:12:24.352190 30669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "325b3dbba80d491dab81840e8eb171e9" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:24.352319 28151 raft_consensus.cc:3037] T 325b3dbba80d491dab81840e8eb171e9 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:24.352365 30669 raft_consensus.cc:3037] T 325b3dbba80d491dab81840e8eb171e9 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:24.354243 30669 raft_consensus.cc:2454] T 325b3dbba80d491dab81840e8eb171e9 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:24.354249 28151 raft_consensus.cc:2454] T 325b3dbba80d491dab81840e8eb171e9 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:24.354646 30737 leader_election.cc:302] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:12:24.354966 32209 raft_consensus.cc:2781] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:24.355155 32209 raft_consensus.cc:686] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Becoming Leader. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Running, Role: LEADER
I20240501 08:12:24.355401 32209 consensus_queue.cc:237] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:24.355906 27617 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:24.356132 27617 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:24.359380 27617 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 759b87b7dafb486b802410fab18bfb11
I20240501 08:12:24.359434 27613 catalog_manager.cc:5484] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64998323a1f4482d8704bf20594d13a5 (127.21.84.135). New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:24.360487 30915 tablet_service.cc:1514] Processing DeleteTablet for tablet 759b87b7dafb486b802410fab18bfb11 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:24 UTC) from {username='slave'} at 127.0.0.1:54036
I20240501 08:12:24.360872 30517 tablet_service.cc:1514] Processing DeleteTablet for tablet 759b87b7dafb486b802410fab18bfb11 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:24 UTC) from {username='slave'} at 127.0.0.1:45078
I20240501 08:12:24.360961 30784 tablet_service.cc:1514] Processing DeleteTablet for tablet 759b87b7dafb486b802410fab18bfb11 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:24 UTC) from {username='slave'} at 127.0.0.1:38952
I20240501 08:12:24.361868 32282 tablet_replica.cc:330] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6: stopping tablet replica
I20240501 08:12:24.362227 32282 raft_consensus.cc:2229] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:24.362607 32282 raft_consensus.cc:2258] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:24.363139 32281 tablet_replica.cc:330] T 759b87b7dafb486b802410fab18bfb11 P a29cd590999b4f32849ff5689cd53c11: stopping tablet replica
I20240501 08:12:24.363888 32282 ts_tablet_manager.cc:1875] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:24.365509 32281 raft_consensus.cc:2229] T 759b87b7dafb486b802410fab18bfb11 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:24.365870 32281 raft_consensus.cc:2258] T 759b87b7dafb486b802410fab18bfb11 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:24.367339 32281 ts_tablet_manager.cc:1875] T 759b87b7dafb486b802410fab18bfb11 P a29cd590999b4f32849ff5689cd53c11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:24.372931 32283 tablet_replica.cc:330] T 759b87b7dafb486b802410fab18bfb11 P 64998323a1f4482d8704bf20594d13a5: stopping tablet replica
I20240501 08:12:24.373281 32283 raft_consensus.cc:2229] T 759b87b7dafb486b802410fab18bfb11 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:24.373634 32283 raft_consensus.cc:2258] T 759b87b7dafb486b802410fab18bfb11 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:24.374120 32282 ts_tablet_manager.cc:1888] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:24.374447 32282 log.cc:1196] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/wals/759b87b7dafb486b802410fab18bfb11
I20240501 08:12:24.374997 32283 ts_tablet_manager.cc:1875] T 759b87b7dafb486b802410fab18bfb11 P 64998323a1f4482d8704bf20594d13a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:24.375025 32282 ts_tablet_manager.cc:1909] T 759b87b7dafb486b802410fab18bfb11 P 31811708c6774037b82df44024b229b6: Deleting consensus metadata
I20240501 08:12:24.375497 32281 ts_tablet_manager.cc:1888] T 759b87b7dafb486b802410fab18bfb11 P a29cd590999b4f32849ff5689cd53c11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:24.375690 32281 log.cc:1196] T 759b87b7dafb486b802410fab18bfb11 P a29cd590999b4f32849ff5689cd53c11: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/wals/759b87b7dafb486b802410fab18bfb11
I20240501 08:12:24.376227 32281 ts_tablet_manager.cc:1909] T 759b87b7dafb486b802410fab18bfb11 P a29cd590999b4f32849ff5689cd53c11: Deleting consensus metadata
I20240501 08:12:24.376847 27607 catalog_manager.cc:4830] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): tablet 759b87b7dafb486b802410fab18bfb11 (table rebalancer_dynamic_table_new_name [id=d4c3fec0dd48477c9d16bc0f521c41e3]) successfully deleted
I20240501 08:12:24.377529 27606 catalog_manager.cc:4830] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): tablet 759b87b7dafb486b802410fab18bfb11 (table rebalancer_dynamic_table_new_name [id=d4c3fec0dd48477c9d16bc0f521c41e3]) successfully deleted
I20240501 08:12:24.381899 32283 ts_tablet_manager.cc:1888] T 759b87b7dafb486b802410fab18bfb11 P 64998323a1f4482d8704bf20594d13a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:24.382061 32283 log.cc:1196] T 759b87b7dafb486b802410fab18bfb11 P 64998323a1f4482d8704bf20594d13a5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal/wals/759b87b7dafb486b802410fab18bfb11
I20240501 08:12:24.382494 32283 ts_tablet_manager.cc:1909] T 759b87b7dafb486b802410fab18bfb11 P 64998323a1f4482d8704bf20594d13a5: Deleting consensus metadata
I20240501 08:12:24.383410 27605 catalog_manager.cc:4830] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): tablet 759b87b7dafb486b802410fab18bfb11 (table rebalancer_dynamic_table_new_name [id=d4c3fec0dd48477c9d16bc0f521c41e3]) successfully deleted
I20240501 08:12:24.463642 27613 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:24.472743 30915 tablet_service.cc:1467] Processing CreateTablet for tablet bcd7af068dc042e6bc0e9336bcf7fb32 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=80a2a4dc9a37486495b4f6a8a08e7554]), partition=
I20240501 08:12:24.472728 27741 tablet_service.cc:1467] Processing CreateTablet for tablet bcd7af068dc042e6bc0e9336bcf7fb32 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=80a2a4dc9a37486495b4f6a8a08e7554]), partition=
I20240501 08:12:24.473124 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcd7af068dc042e6bc0e9336bcf7fb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:24.473150 30915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcd7af068dc042e6bc0e9336bcf7fb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:24.474598 30652 tablet_service.cc:1467] Processing CreateTablet for tablet bcd7af068dc042e6bc0e9336bcf7fb32 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=80a2a4dc9a37486495b4f6a8a08e7554]), partition=
I20240501 08:12:24.475035 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcd7af068dc042e6bc0e9336bcf7fb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:24.488626 32285 tablet_bootstrap.cc:492] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:24.490247 32261 tablet_bootstrap.cc:492] T bcd7af068dc042e6bc0e9336bcf7fb32 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:24.491786 32261 tablet_bootstrap.cc:654] T bcd7af068dc042e6bc0e9336bcf7fb32 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:24.491786 32285 tablet_bootstrap.cc:654] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:24.493171 32284 tablet_bootstrap.cc:492] T bcd7af068dc042e6bc0e9336bcf7fb32 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:24.494947 32284 tablet_bootstrap.cc:654] T bcd7af068dc042e6bc0e9336bcf7fb32 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:24.506620 32284 tablet_bootstrap.cc:492] T bcd7af068dc042e6bc0e9336bcf7fb32 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:12:24.506752 32261 tablet_bootstrap.cc:492] T bcd7af068dc042e6bc0e9336bcf7fb32 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:24.506788 32284 ts_tablet_manager.cc:1367] T bcd7af068dc042e6bc0e9336bcf7fb32 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20240501 08:12:24.506909 32261 ts_tablet_manager.cc:1367] T bcd7af068dc042e6bc0e9336bcf7fb32 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.008s
I20240501 08:12:24.507457 32284 raft_consensus.cc:348] T bcd7af068dc042e6bc0e9336bcf7fb32 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:24.507495 32261 raft_consensus.cc:348] T bcd7af068dc042e6bc0e9336bcf7fb32 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:24.507666 32261 raft_consensus.cc:374] T bcd7af068dc042e6bc0e9336bcf7fb32 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:24.507684 32284 raft_consensus.cc:374] T bcd7af068dc042e6bc0e9336bcf7fb32 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:24.507757 32284 raft_consensus.cc:729] T bcd7af068dc042e6bc0e9336bcf7fb32 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:12:24.507751 32261 raft_consensus.cc:729] T bcd7af068dc042e6bc0e9336bcf7fb32 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:24.507951 32284 consensus_queue.cc:260] T bcd7af068dc042e6bc0e9336bcf7fb32 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:24.508227 32261 consensus_queue.cc:260] T bcd7af068dc042e6bc0e9336bcf7fb32 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:24.508826 32261 ts_tablet_manager.cc:1398] T bcd7af068dc042e6bc0e9336bcf7fb32 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:24.509420 32284 ts_tablet_manager.cc:1398] T bcd7af068dc042e6bc0e9336bcf7fb32 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:24.510056 32285 tablet_bootstrap.cc:492] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:24.510258 32285 ts_tablet_manager.cc:1367] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.010s
I20240501 08:12:24.511672 32285 raft_consensus.cc:348] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:24.511946 32285 raft_consensus.cc:374] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:24.512053 32285 raft_consensus.cc:729] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:24.512295 32285 consensus_queue.cc:260] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:24.512871 32285 ts_tablet_manager.cc:1398] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:24.612373 32217 raft_consensus.cc:482] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:24.612562 32217 raft_consensus.cc:504] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:24.613462 32217 leader_election.cc:288] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:24.614022 30669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd7af068dc042e6bc0e9336bcf7fb32" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:24.614171 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd7af068dc042e6bc0e9336bcf7fb32" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:12:24.614293 30669 raft_consensus.cc:2454] T bcd7af068dc042e6bc0e9336bcf7fb32 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:24.614380 27761 raft_consensus.cc:2454] T bcd7af068dc042e6bc0e9336bcf7fb32 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:24.614738 30870 leader_election.cc:302] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a022e1a501f4d44a0468054d6c3de47, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:24.615036 32217 raft_consensus.cc:2781] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:24.615149 32217 raft_consensus.cc:482] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:24.615237 32217 raft_consensus.cc:3037] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:24.617058 32217 raft_consensus.cc:504] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:24.617625 32217 leader_election.cc:288] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Requested vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:24.618103 30669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd7af068dc042e6bc0e9336bcf7fb32" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:24.618104 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd7af068dc042e6bc0e9336bcf7fb32" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:12:24.618290 27761 raft_consensus.cc:3037] T bcd7af068dc042e6bc0e9336bcf7fb32 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:24.618295 30669 raft_consensus.cc:3037] T bcd7af068dc042e6bc0e9336bcf7fb32 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:24.620203 30669 raft_consensus.cc:2454] T bcd7af068dc042e6bc0e9336bcf7fb32 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:24.620335 27761 raft_consensus.cc:2454] T bcd7af068dc042e6bc0e9336bcf7fb32 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:24.620687 30870 leader_election.cc:302] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a022e1a501f4d44a0468054d6c3de47, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:24.621065 32217 raft_consensus.cc:2781] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:24.621210 32217 raft_consensus.cc:686] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Becoming Leader. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:12:24.621474 32217 consensus_queue.cc:237] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:24.624902 27613 catalog_manager.cc:5484] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: term changed from 0 to 1, leader changed from <none> to a29cd590999b4f32849ff5689cd53c11 (127.21.84.136). New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:24.717665 32209 consensus_queue.cc:1035] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:24.721145 32273 consensus_queue.cc:1035] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:24.733918 27613 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:24.740825 30669 raft_consensus.cc:1261] T bcd7af068dc042e6bc0e9336bcf7fb32 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:24.740906 27761 raft_consensus.cc:1261] T bcd7af068dc042e6bc0e9336bcf7fb32 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:24.742219 32217 consensus_queue.cc:1035] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:24.742556 32214 consensus_queue.cc:1035] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:24.754329 32256 tablet.cc:1722] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:24.754477 27613 catalog_manager.cc:5576] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471) does not have the latest schema for tablet bcd7af068dc042e6bc0e9336bcf7fb32 (table rebalancer_dynamic_table [id=80a2a4dc9a37486495b4f6a8a08e7554]). Expected version 1 got 0
I20240501 08:12:24.754917 32256 tablet.cc:1620] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11: MemRowSet was empty: no flush needed.
I20240501 08:12:24.757042 27613 catalog_manager.cc:5576] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933) does not have the latest schema for tablet bcd7af068dc042e6bc0e9336bcf7fb32 (table rebalancer_dynamic_table [id=80a2a4dc9a37486495b4f6a8a08e7554]). Expected version 1 got 0
I20240501 08:12:24.758678 32298 tablet.cc:1722] T bcd7af068dc042e6bc0e9336bcf7fb32 P 6331f77dae9d452184e8cd0cbf5f1b43: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:24.759352 32298 tablet.cc:1620] T bcd7af068dc042e6bc0e9336bcf7fb32 P 6331f77dae9d452184e8cd0cbf5f1b43: MemRowSet was empty: no flush needed.
I20240501 08:12:24.760927 27617 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=80a2a4dc9a37486495b4f6a8a08e7554] alter complete (version 1)
I20240501 08:12:24.767481 32299 tablet.cc:1722] T bcd7af068dc042e6bc0e9336bcf7fb32 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:24.768004 32299 tablet.cc:1620] T bcd7af068dc042e6bc0e9336bcf7fb32 P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:24.771163 27620 catalog_manager.cc:5576] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933) does not have the latest schema for tablet bcd7af068dc042e6bc0e9336bcf7fb32 (table rebalancer_dynamic_table [id=80a2a4dc9a37486495b4f6a8a08e7554]). Expected version 1 got 0
I20240501 08:12:24.816140 32273 consensus_queue.cc:1035] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:24.821514 32209 consensus_queue.cc:1035] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:24.839170 32218 consensus_queue.cc:1035] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:24.847959 32307 consensus_queue.cc:1035] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:24.866637 27620 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:24.873670 32256 tablet.cc:1722] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:24.874339 32256 tablet.cc:1620] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11: MemRowSet was empty: no flush needed.
I20240501 08:12:24.874608 32298 tablet.cc:1722] T bcd7af068dc042e6bc0e9336bcf7fb32 P 6331f77dae9d452184e8cd0cbf5f1b43: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:24.874881 32299 tablet.cc:1722] T bcd7af068dc042e6bc0e9336bcf7fb32 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:24.875074 32298 tablet.cc:1620] T bcd7af068dc042e6bc0e9336bcf7fb32 P 6331f77dae9d452184e8cd0cbf5f1b43: MemRowSet was empty: no flush needed.
I20240501 08:12:24.875358 32299 tablet.cc:1620] T bcd7af068dc042e6bc0e9336bcf7fb32 P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:24.884253 27616 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_extra_table_44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:24.888931 27617 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=80a2a4dc9a37486495b4f6a8a08e7554] alter complete (version 2)
I20240501 08:12:24.894289 30915 tablet_service.cc:1467] Processing CreateTablet for tablet 73058afba1214e21b655283c0612a0ec (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=a727b21f159445ba8749f9bf1bec2b3d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:24.894482 30784 tablet_service.cc:1467] Processing CreateTablet for tablet 73058afba1214e21b655283c0612a0ec (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=a727b21f159445ba8749f9bf1bec2b3d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:24.894829 30915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73058afba1214e21b655283c0612a0ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:24.894943 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73058afba1214e21b655283c0612a0ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:24.894930 30517 tablet_service.cc:1467] Processing CreateTablet for tablet 4f8c0a8e86f04f32b79743f819948544 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=a727b21f159445ba8749f9bf1bec2b3d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:24.895331 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f8c0a8e86f04f32b79743f819948544. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:24.896080 30514 tablet_service.cc:1467] Processing CreateTablet for tablet f63215605ade41a5bc8d97a35ce33a31 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=a727b21f159445ba8749f9bf1bec2b3d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:24.896454 30514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63215605ade41a5bc8d97a35ce33a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:24.900619 28002 tablet_service.cc:1467] Processing CreateTablet for tablet 4f8c0a8e86f04f32b79743f819948544 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=a727b21f159445ba8749f9bf1bec2b3d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:24.900614 30652 tablet_service.cc:1467] Processing CreateTablet for tablet 73058afba1214e21b655283c0612a0ec (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=a727b21f159445ba8749f9bf1bec2b3d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:24.901038 28002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f8c0a8e86f04f32b79743f819948544. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:24.901069 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73058afba1214e21b655283c0612a0ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:24.902276 32285 tablet_bootstrap.cc:492] T 73058afba1214e21b655283c0612a0ec P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:24.903915 32285 tablet_bootstrap.cc:654] T 73058afba1214e21b655283c0612a0ec P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:24.907830 32310 tablet_bootstrap.cc:492] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:24.908550 32261 tablet_bootstrap.cc:492] T 73058afba1214e21b655283c0612a0ec P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:24.909015 30916 tablet_service.cc:1467] Processing CreateTablet for tablet f63215605ade41a5bc8d97a35ce33a31 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=a727b21f159445ba8749f9bf1bec2b3d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:24.909464 30916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63215605ade41a5bc8d97a35ce33a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:24.910282 32310 tablet_bootstrap.cc:654] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:24.910462 32261 tablet_bootstrap.cc:654] T 73058afba1214e21b655283c0612a0ec P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:24.917721 28000 tablet_service.cc:1467] Processing CreateTablet for tablet f63215605ade41a5bc8d97a35ce33a31 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=a727b21f159445ba8749f9bf1bec2b3d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:24.918764 28000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63215605ade41a5bc8d97a35ce33a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:24.927387 30915 tablet_service.cc:1467] Processing CreateTablet for tablet 4f8c0a8e86f04f32b79743f819948544 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=a727b21f159445ba8749f9bf1bec2b3d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:24.928068 30915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f8c0a8e86f04f32b79743f819948544. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:24.938294 32285 tablet_bootstrap.cc:492] T 73058afba1214e21b655283c0612a0ec P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:24.939260 32315 tablet_bootstrap.cc:492] T 4f8c0a8e86f04f32b79743f819948544 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:24.939661 32285 ts_tablet_manager.cc:1367] T 73058afba1214e21b655283c0612a0ec P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.003s
I20240501 08:12:24.941071 32285 raft_consensus.cc:348] T 73058afba1214e21b655283c0612a0ec P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:24.942230 32285 raft_consensus.cc:374] T 73058afba1214e21b655283c0612a0ec P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:24.942442 32285 raft_consensus.cc:729] T 73058afba1214e21b655283c0612a0ec P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:24.942982 32314 tablet_bootstrap.cc:492] T 73058afba1214e21b655283c0612a0ec P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:24.943087 32261 tablet_bootstrap.cc:492] T 73058afba1214e21b655283c0612a0ec P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:24.943310 32261 ts_tablet_manager.cc:1367] T 73058afba1214e21b655283c0612a0ec P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.035s user 0.007s sys 0.004s
I20240501 08:12:24.943470 32285 consensus_queue.cc:260] T 73058afba1214e21b655283c0612a0ec P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:24.944092 32285 ts_tablet_manager.cc:1398] T 73058afba1214e21b655283c0612a0ec P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:24.944334 32261 raft_consensus.cc:348] T 73058afba1214e21b655283c0612a0ec P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:24.944607 32261 raft_consensus.cc:374] T 73058afba1214e21b655283c0612a0ec P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:24.945261 32261 raft_consensus.cc:729] T 73058afba1214e21b655283c0612a0ec P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:24.945591 32261 consensus_queue.cc:260] T 73058afba1214e21b655283c0612a0ec P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:24.946362 32261 ts_tablet_manager.cc:1398] T 73058afba1214e21b655283c0612a0ec P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:12:24.947495 32315 tablet_bootstrap.cc:654] T 4f8c0a8e86f04f32b79743f819948544 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:24.949800 32314 tablet_bootstrap.cc:654] T 73058afba1214e21b655283c0612a0ec P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:24.949864 32285 tablet_bootstrap.cc:492] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:24.960301 32285 tablet_bootstrap.cc:654] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:24.965560 32315 tablet_bootstrap.cc:492] T 4f8c0a8e86f04f32b79743f819948544 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:12:24.965911 32315 ts_tablet_manager.cc:1367] T 4f8c0a8e86f04f32b79743f819948544 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.007s
I20240501 08:12:24.966742 32315 raft_consensus.cc:348] T 4f8c0a8e86f04f32b79743f819948544 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:24.967555 32315 raft_consensus.cc:374] T 4f8c0a8e86f04f32b79743f819948544 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:24.967710 32315 raft_consensus.cc:729] T 4f8c0a8e86f04f32b79743f819948544 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:12:24.968005 32315 consensus_queue.cc:260] T 4f8c0a8e86f04f32b79743f819948544 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:24.970337 32315 ts_tablet_manager.cc:1398] T 4f8c0a8e86f04f32b79743f819948544 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 08:12:24.970705 32315 tablet_bootstrap.cc:492] T f63215605ade41a5bc8d97a35ce33a31 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:24.976267 32315 tablet_bootstrap.cc:654] T f63215605ade41a5bc8d97a35ce33a31 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:24.985733 32315 tablet_bootstrap.cc:492] T f63215605ade41a5bc8d97a35ce33a31 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:12:24.986068 32315 ts_tablet_manager.cc:1367] T f63215605ade41a5bc8d97a35ce33a31 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.006s
I20240501 08:12:24.986922 32315 raft_consensus.cc:348] T f63215605ade41a5bc8d97a35ce33a31 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:24.989187 32315 raft_consensus.cc:374] T f63215605ade41a5bc8d97a35ce33a31 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:24.989331 32315 raft_consensus.cc:729] T f63215605ade41a5bc8d97a35ce33a31 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:12:24.990216 32315 consensus_queue.cc:260] T f63215605ade41a5bc8d97a35ce33a31 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:24.990411 32285 tablet_bootstrap.cc:492] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:24.990692 32285 ts_tablet_manager.cc:1367] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.045s user 0.000s sys 0.009s
I20240501 08:12:24.991796 32285 raft_consensus.cc:348] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:24.992439 32315 ts_tablet_manager.cc:1398] T f63215605ade41a5bc8d97a35ce33a31 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240501 08:12:24.994753 32285 raft_consensus.cc:374] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:24.994999 32285 raft_consensus.cc:729] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:24.995800 32285 consensus_queue.cc:260] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:24.997910 32285 ts_tablet_manager.cc:1398] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240501 08:12:25.001533 32285 tablet_bootstrap.cc:492] T 4f8c0a8e86f04f32b79743f819948544 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:25.003057 32310 tablet_bootstrap.cc:492] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:25.003265 32310 ts_tablet_manager.cc:1367] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.096s user 0.024s sys 0.014s
I20240501 08:12:25.004197 32310 raft_consensus.cc:348] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:25.004456 32310 raft_consensus.cc:374] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:25.004551 32310 raft_consensus.cc:729] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:25.004855 32310 consensus_queue.cc:260] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:25.006932 32314 tablet_bootstrap.cc:492] T 73058afba1214e21b655283c0612a0ec P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:25.007069 32310 ts_tablet_manager.cc:1398] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 08:12:25.007229 32314 ts_tablet_manager.cc:1367] T 73058afba1214e21b655283c0612a0ec P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.064s user 0.025s sys 0.021s
I20240501 08:12:25.007447 32310 tablet_bootstrap.cc:492] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:25.008245 32314 raft_consensus.cc:348] T 73058afba1214e21b655283c0612a0ec P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:25.009271 32285 tablet_bootstrap.cc:654] T 4f8c0a8e86f04f32b79743f819948544 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:25.009845 32314 raft_consensus.cc:374] T 73058afba1214e21b655283c0612a0ec P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:25.010046 32314 raft_consensus.cc:729] T 73058afba1214e21b655283c0612a0ec P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:25.010408 32314 consensus_queue.cc:260] T 73058afba1214e21b655283c0612a0ec P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:25.010856 32310 tablet_bootstrap.cc:654] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:25.013365 32314 ts_tablet_manager.cc:1398] T 73058afba1214e21b655283c0612a0ec P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240501 08:12:25.022822 32310 tablet_bootstrap.cc:492] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:25.022992 32310 ts_tablet_manager.cc:1367] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.001s
I20240501 08:12:25.023732 32310 raft_consensus.cc:348] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:25.024313 32310 raft_consensus.cc:374] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:25.024294 27617 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:25.024431 32310 raft_consensus.cc:729] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:25.024444 32285 tablet_bootstrap.cc:492] T 4f8c0a8e86f04f32b79743f819948544 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:25.024807 32310 consensus_queue.cc:260] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:25.025086 32285 ts_tablet_manager.cc:1367] T 4f8c0a8e86f04f32b79743f819948544 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.004s
I20240501 08:12:25.026337 32285 raft_consensus.cc:348] T 4f8c0a8e86f04f32b79743f819948544 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:25.026626 32285 raft_consensus.cc:374] T 4f8c0a8e86f04f32b79743f819948544 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:25.026633 32310 ts_tablet_manager.cc:1398] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 08:12:25.026741 32285 raft_consensus.cc:729] T 4f8c0a8e86f04f32b79743f819948544 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:25.027022 32285 consensus_queue.cc:260] T 4f8c0a8e86f04f32b79743f819948544 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:25.027592 32285 ts_tablet_manager.cc:1398] T 4f8c0a8e86f04f32b79743f819948544 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:25.035243 32256 tablet.cc:1722] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:25.035585 32299 tablet.cc:1722] T bcd7af068dc042e6bc0e9336bcf7fb32 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:25.045640 32298 tablet.cc:1722] T bcd7af068dc042e6bc0e9336bcf7fb32 P 6331f77dae9d452184e8cd0cbf5f1b43: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:25.054953 27616 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table_new_name [id=80a2a4dc9a37486495b4f6a8a08e7554] alter complete (version 3)
I20240501 08:12:25.155072 32228 raft_consensus.cc:482] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:25.155264 32228 raft_consensus.cc:504] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:25.155817 32228 leader_election.cc:288] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:25.156332 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f8c0a8e86f04f32b79743f819948544" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:12:25.156421 30935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f8c0a8e86f04f32b79743f819948544" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:25.156544 28022 raft_consensus.cc:2454] T 4f8c0a8e86f04f32b79743f819948544 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 0.
I20240501 08:12:25.156652 30935 raft_consensus.cc:2454] T 4f8c0a8e86f04f32b79743f819948544 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 0.
I20240501 08:12:25.156997 30472 leader_election.cc:302] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:12:25.157271 32228 raft_consensus.cc:2781] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:25.157382 32228 raft_consensus.cc:482] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:25.158035 32228 raft_consensus.cc:3037] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:25.158316 27616 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:25.158512 27616 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:25.161870 27616 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet bcd7af068dc042e6bc0e9336bcf7fb32
I20240501 08:12:25.162200 32228 raft_consensus.cc:504] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:25.162637 30915 tablet_service.cc:1514] Processing DeleteTablet for tablet bcd7af068dc042e6bc0e9336bcf7fb32 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:25 UTC) from {username='slave'} at 127.0.0.1:54036
I20240501 08:12:25.162829 30652 tablet_service.cc:1514] Processing DeleteTablet for tablet bcd7af068dc042e6bc0e9336bcf7fb32 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:25 UTC) from {username='slave'} at 127.0.0.1:57156
I20240501 08:12:25.162865 27741 tablet_service.cc:1514] Processing DeleteTablet for tablet bcd7af068dc042e6bc0e9336bcf7fb32 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:25 UTC) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:25.163460 32323 tablet_replica.cc:330] T bcd7af068dc042e6bc0e9336bcf7fb32 P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:25.163766 32323 raft_consensus.cc:2229] T bcd7af068dc042e6bc0e9336bcf7fb32 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:25.164079 32323 raft_consensus.cc:2258] T bcd7af068dc042e6bc0e9336bcf7fb32 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:25.164104 32322 tablet_replica.cc:330] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11: stopping tablet replica
I20240501 08:12:25.164912 32322 raft_consensus.cc:2229] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:25.165166 32323 ts_tablet_manager.cc:1875] T bcd7af068dc042e6bc0e9336bcf7fb32 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:25.165452 32322 raft_consensus.cc:2258] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:25.165534 32324 tablet_replica.cc:330] T bcd7af068dc042e6bc0e9336bcf7fb32 P 9a022e1a501f4d44a0468054d6c3de47: stopping tablet replica
I20240501 08:12:25.166193 32324 raft_consensus.cc:2229] T bcd7af068dc042e6bc0e9336bcf7fb32 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:25.166342 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f8c0a8e86f04f32b79743f819948544" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:12:25.166476 32324 raft_consensus.cc:2258] T bcd7af068dc042e6bc0e9336bcf7fb32 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:25.166465 30935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f8c0a8e86f04f32b79743f819948544" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:25.166594 28022 raft_consensus.cc:3037] T 4f8c0a8e86f04f32b79743f819948544 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:25.166632 30935 raft_consensus.cc:3037] T 4f8c0a8e86f04f32b79743f819948544 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:25.167369 32324 ts_tablet_manager.cc:1875] T bcd7af068dc042e6bc0e9336bcf7fb32 P 9a022e1a501f4d44a0468054d6c3de47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:25.169394 30935 raft_consensus.cc:2454] T 4f8c0a8e86f04f32b79743f819948544 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 1.
I20240501 08:12:25.169395 28022 raft_consensus.cc:2454] T 4f8c0a8e86f04f32b79743f819948544 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 1.
I20240501 08:12:25.169816 30472 leader_election.cc:302] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:25.171962 32322 ts_tablet_manager.cc:1875] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:25.172652 32228 leader_election.cc:288] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 election: Requested vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:25.172852 32228 raft_consensus.cc:2781] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:25.172991 32228 raft_consensus.cc:686] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Becoming Leader. State: Replica: 31811708c6774037b82df44024b229b6, State: Running, Role: LEADER
I20240501 08:12:25.173918 32228 consensus_queue.cc:237] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:25.175307 32324 ts_tablet_manager.cc:1888] T bcd7af068dc042e6bc0e9336bcf7fb32 P 9a022e1a501f4d44a0468054d6c3de47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:25.175416 32323 ts_tablet_manager.cc:1888] T bcd7af068dc042e6bc0e9336bcf7fb32 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:25.175490 32324 log.cc:1196] T bcd7af068dc042e6bc0e9336bcf7fb32 P 9a022e1a501f4d44a0468054d6c3de47: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/wals/bcd7af068dc042e6bc0e9336bcf7fb32
I20240501 08:12:25.175590 32323 log.cc:1196] T bcd7af068dc042e6bc0e9336bcf7fb32 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/bcd7af068dc042e6bc0e9336bcf7fb32
I20240501 08:12:25.176081 32324 ts_tablet_manager.cc:1909] T bcd7af068dc042e6bc0e9336bcf7fb32 P 9a022e1a501f4d44a0468054d6c3de47: Deleting consensus metadata
I20240501 08:12:25.176088 32323 ts_tablet_manager.cc:1909] T bcd7af068dc042e6bc0e9336bcf7fb32 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting consensus metadata
I20240501 08:12:25.177130 27604 catalog_manager.cc:4830] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): tablet bcd7af068dc042e6bc0e9336bcf7fb32 (table rebalancer_dynamic_table_new_name [id=80a2a4dc9a37486495b4f6a8a08e7554]) successfully deleted
I20240501 08:12:25.177245 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet bcd7af068dc042e6bc0e9336bcf7fb32 (table rebalancer_dynamic_table_new_name [id=80a2a4dc9a37486495b4f6a8a08e7554]) successfully deleted
I20240501 08:12:25.181412 27617 catalog_manager.cc:5484] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31811708c6774037b82df44024b229b6 (127.21.84.133). New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:25.182935 32322 ts_tablet_manager.cc:1888] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:25.183082 32322 log.cc:1196] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/wals/bcd7af068dc042e6bc0e9336bcf7fb32
I20240501 08:12:25.183557 32322 ts_tablet_manager.cc:1909] T bcd7af068dc042e6bc0e9336bcf7fb32 P a29cd590999b4f32849ff5689cd53c11: Deleting consensus metadata
I20240501 08:12:25.184701 27606 catalog_manager.cc:4830] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): tablet bcd7af068dc042e6bc0e9336bcf7fb32 (table rebalancer_dynamic_table_new_name [id=80a2a4dc9a37486495b4f6a8a08e7554]) successfully deleted
I20240501 08:12:25.265051 27617 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:25.275549 30915 tablet_service.cc:1467] Processing CreateTablet for tablet eaadc7787c3f439b9457f8092df745b3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5ac50e5b3c3443c5b510ee241fbdf469]), partition=
I20240501 08:12:25.275597 30517 tablet_service.cc:1467] Processing CreateTablet for tablet eaadc7787c3f439b9457f8092df745b3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5ac50e5b3c3443c5b510ee241fbdf469]), partition=
I20240501 08:12:25.275934 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaadc7787c3f439b9457f8092df745b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:25.275935 30915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaadc7787c3f439b9457f8092df745b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:25.280911 32285 tablet_bootstrap.cc:492] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:25.281217 30652 tablet_service.cc:1467] Processing CreateTablet for tablet eaadc7787c3f439b9457f8092df745b3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5ac50e5b3c3443c5b510ee241fbdf469]), partition=
I20240501 08:12:25.281560 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaadc7787c3f439b9457f8092df745b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:25.282706 32285 tablet_bootstrap.cc:654] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:25.284926 32310 tablet_bootstrap.cc:492] T eaadc7787c3f439b9457f8092df745b3 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:25.286245 32310 tablet_bootstrap.cc:654] T eaadc7787c3f439b9457f8092df745b3 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:25.288569 32310 tablet_bootstrap.cc:492] T eaadc7787c3f439b9457f8092df745b3 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:25.288717 32310 ts_tablet_manager.cc:1367] T eaadc7787c3f439b9457f8092df745b3 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:12:25.290046 32310 raft_consensus.cc:348] T eaadc7787c3f439b9457f8092df745b3 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:25.290241 32310 raft_consensus.cc:374] T eaadc7787c3f439b9457f8092df745b3 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:25.290325 32310 raft_consensus.cc:729] T eaadc7787c3f439b9457f8092df745b3 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:25.290526 32261 tablet_bootstrap.cc:492] T eaadc7787c3f439b9457f8092df745b3 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:25.290547 32310 consensus_queue.cc:260] T eaadc7787c3f439b9457f8092df745b3 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:25.291035 32310 ts_tablet_manager.cc:1398] T eaadc7787c3f439b9457f8092df745b3 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:25.292080 32261 tablet_bootstrap.cc:654] T eaadc7787c3f439b9457f8092df745b3 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:25.292834 32285 tablet_bootstrap.cc:492] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:25.292966 32285 ts_tablet_manager.cc:1367] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.003s
I20240501 08:12:25.293835 32285 raft_consensus.cc:348] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:25.294348 32285 raft_consensus.cc:374] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:25.294478 32285 raft_consensus.cc:729] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:25.294824 32285 consensus_queue.cc:260] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:25.295418 32285 ts_tablet_manager.cc:1398] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:25.296626 32261 tablet_bootstrap.cc:492] T eaadc7787c3f439b9457f8092df745b3 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:25.296780 32261 ts_tablet_manager.cc:1367] T eaadc7787c3f439b9457f8092df745b3 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20240501 08:12:25.297389 32261 raft_consensus.cc:348] T eaadc7787c3f439b9457f8092df745b3 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:25.297623 32261 raft_consensus.cc:374] T eaadc7787c3f439b9457f8092df745b3 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:25.297739 32261 raft_consensus.cc:729] T eaadc7787c3f439b9457f8092df745b3 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:25.297993 32261 consensus_queue.cc:260] T eaadc7787c3f439b9457f8092df745b3 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:25.298595 32261 ts_tablet_manager.cc:1398] T eaadc7787c3f439b9457f8092df745b3 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:25.317025 32309 raft_consensus.cc:482] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:25.317209 32309 raft_consensus.cc:504] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:25.318148 32309 leader_election.cc:288] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:25.318606 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63215605ade41a5bc8d97a35ce33a31" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:25.318922 30537 raft_consensus.cc:2454] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:25.318996 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63215605ade41a5bc8d97a35ce33a31" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:12:25.319532 28022 raft_consensus.cc:2454] T f63215605ade41a5bc8d97a35ce33a31 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:25.319880 30873 leader_election.cc:302] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:25.320428 32220 raft_consensus.cc:2781] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:25.320753 32220 raft_consensus.cc:482] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:25.320958 32220 raft_consensus.cc:3037] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:25.324396 32220 raft_consensus.cc:504] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:25.324959 32220 leader_election.cc:288] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Requested vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:25.325464 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63215605ade41a5bc8d97a35ce33a31" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:25.325718 30536 raft_consensus.cc:3037] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:25.325704 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63215605ade41a5bc8d97a35ce33a31" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:12:25.325891 28022 raft_consensus.cc:3037] T f63215605ade41a5bc8d97a35ce33a31 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:25.327746 30536 raft_consensus.cc:2454] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:25.328143 28022 raft_consensus.cc:2454] T f63215605ade41a5bc8d97a35ce33a31 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:25.328474 30873 leader_election.cc:302] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:25.328756 32214 raft_consensus.cc:2781] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:25.328899 32214 raft_consensus.cc:686] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Becoming Leader. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:12:25.329222 32214 consensus_queue.cc:237] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:25.333297 27616 catalog_manager.cc:5484] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: term changed from 0 to 1, leader changed from <none> to a29cd590999b4f32849ff5689cd53c11 (127.21.84.136). New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:25.384611 32214 raft_consensus.cc:482] T 73058afba1214e21b655283c0612a0ec P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:25.384786 32214 raft_consensus.cc:504] T 73058afba1214e21b655283c0612a0ec P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:25.385444 32214 leader_election.cc:288] T 73058afba1214e21b655283c0612a0ec P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:25.385993 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73058afba1214e21b655283c0612a0ec" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:25.386243 30804 raft_consensus.cc:2454] T 73058afba1214e21b655283c0612a0ec P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:25.386200 30669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73058afba1214e21b655283c0612a0ec" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:25.386443 30669 raft_consensus.cc:2454] T 73058afba1214e21b655283c0612a0ec P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:25.386819 30871 leader_election.cc:302] T 73058afba1214e21b655283c0612a0ec P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:25.387462 32220 raft_consensus.cc:2781] T 73058afba1214e21b655283c0612a0ec P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:25.387555 32220 raft_consensus.cc:482] T 73058afba1214e21b655283c0612a0ec P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:25.387673 32220 raft_consensus.cc:3037] T 73058afba1214e21b655283c0612a0ec P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:25.390733 32220 raft_consensus.cc:504] T 73058afba1214e21b655283c0612a0ec P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:25.391427 32220 leader_election.cc:288] T 73058afba1214e21b655283c0612a0ec P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Requested vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:25.391829 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73058afba1214e21b655283c0612a0ec" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:25.392076 30804 raft_consensus.cc:3037] T 73058afba1214e21b655283c0612a0ec P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:25.392385 30669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73058afba1214e21b655283c0612a0ec" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:25.392604 30669 raft_consensus.cc:3037] T 73058afba1214e21b655283c0612a0ec P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:25.395376 30804 raft_consensus.cc:2454] T 73058afba1214e21b655283c0612a0ec P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:25.396219 30871 leader_election.cc:302] T 73058afba1214e21b655283c0612a0ec P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:25.396508 32214 raft_consensus.cc:2781] T 73058afba1214e21b655283c0612a0ec P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:25.396601 30669 raft_consensus.cc:2454] T 73058afba1214e21b655283c0612a0ec P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:25.396735 32214 raft_consensus.cc:686] T 73058afba1214e21b655283c0612a0ec P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Becoming Leader. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:12:25.397022 32214 consensus_queue.cc:237] T 73058afba1214e21b655283c0612a0ec P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:25.403676 27616 catalog_manager.cc:5484] T 73058afba1214e21b655283c0612a0ec P a29cd590999b4f32849ff5689cd53c11 reported cstate change: term changed from 0 to 1, leader changed from <none> to a29cd590999b4f32849ff5689cd53c11 (127.21.84.136). New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:25.404814 32214 raft_consensus.cc:482] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:25.404959 32214 raft_consensus.cc:504] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:25.406121 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaadc7787c3f439b9457f8092df745b3" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:25.405632 32214 leader_election.cc:288] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:25.406208 30669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaadc7787c3f439b9457f8092df745b3" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:25.406381 30536 raft_consensus.cc:2454] T eaadc7787c3f439b9457f8092df745b3 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:25.406422 30669 raft_consensus.cc:2454] T eaadc7787c3f439b9457f8092df745b3 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:25.406827 30870 leader_election.cc:302] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 9a022e1a501f4d44a0468054d6c3de47, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:25.407217 32214 raft_consensus.cc:2781] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:25.407328 32214 raft_consensus.cc:482] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:25.407389 32214 raft_consensus.cc:3037] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:25.409272 32214 raft_consensus.cc:504] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:25.409893 32214 leader_election.cc:288] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Requested vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:25.410313 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaadc7787c3f439b9457f8092df745b3" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:25.410513 30536 raft_consensus.cc:3037] T eaadc7787c3f439b9457f8092df745b3 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:25.410542 30669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaadc7787c3f439b9457f8092df745b3" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:25.410727 30669 raft_consensus.cc:3037] T eaadc7787c3f439b9457f8092df745b3 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:25.413334 30536 raft_consensus.cc:2454] T eaadc7787c3f439b9457f8092df745b3 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:25.413354 30669 raft_consensus.cc:2454] T eaadc7787c3f439b9457f8092df745b3 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:25.413839 30873 leader_election.cc:302] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:25.414316 32214 raft_consensus.cc:2781] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:25.414844 32214 raft_consensus.cc:686] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Becoming Leader. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:12:25.415660 32214 consensus_queue.cc:237] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:25.424619 27616 catalog_manager.cc:5484] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: term changed from 0 to 1, leader changed from <none> to a29cd590999b4f32849ff5689cd53c11 (127.21.84.136). New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:25.546330 27616 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:25.569363 32228 consensus_queue.cc:1035] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:25.573383 30669 raft_consensus.cc:1261] T eaadc7787c3f439b9457f8092df745b3 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:25.574013 32309 consensus_queue.cc:1035] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:25.574115 30536 raft_consensus.cc:1261] T eaadc7787c3f439b9457f8092df745b3 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:25.575614 32309 consensus_queue.cc:1035] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:25.582615 27616 catalog_manager.cc:5576] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293) does not have the latest schema for tablet eaadc7787c3f439b9457f8092df745b3 (table rebalancer_dynamic_table [id=5ac50e5b3c3443c5b510ee241fbdf469]). Expected version 1 got 0
I20240501 08:12:25.587891 32339 tablet.cc:1722] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:25.588497 32339 tablet.cc:1620] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11: MemRowSet was empty: no flush needed.
I20240501 08:12:25.589097 32336 mvcc.cc:205] Tried to move back new op lower bound from 7022801492261797888 to 7022801491627606016. Current Snapshot: MvccSnapshot[applied={T|T < 7022801492261797888}]
I20240501 08:12:25.591389 27613 catalog_manager.cc:5576] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845) does not have the latest schema for tablet eaadc7787c3f439b9457f8092df745b3 (table rebalancer_dynamic_table [id=5ac50e5b3c3443c5b510ee241fbdf469]). Expected version 1 got 0
I20240501 08:12:25.595212 32340 tablet.cc:1722] T eaadc7787c3f439b9457f8092df745b3 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:25.595830 32340 tablet.cc:1620] T eaadc7787c3f439b9457f8092df745b3 P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:25.598137 27613 catalog_manager.cc:5576] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293) does not have the latest schema for tablet eaadc7787c3f439b9457f8092df745b3 (table rebalancer_dynamic_table [id=5ac50e5b3c3443c5b510ee241fbdf469]). Expected version 1 got 0
I20240501 08:12:25.607895 32228 consensus_queue.cc:1035] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:25.618152 27613 catalog_manager.cc:5576] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933) does not have the latest schema for tablet eaadc7787c3f439b9457f8092df745b3 (table rebalancer_dynamic_table [id=5ac50e5b3c3443c5b510ee241fbdf469]). Expected version 1 got 0
I20240501 08:12:25.618602 27616 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=5ac50e5b3c3443c5b510ee241fbdf469] alter complete (version 1)
I20240501 08:12:25.624768 32343 tablet.cc:1722] T eaadc7787c3f439b9457f8092df745b3 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:25.626271 32343 tablet.cc:1620] T eaadc7787c3f439b9457f8092df745b3 P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:25.629158 27616 catalog_manager.cc:5576] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933) does not have the latest schema for tablet eaadc7787c3f439b9457f8092df745b3 (table rebalancer_dynamic_table [id=5ac50e5b3c3443c5b510ee241fbdf469]). Expected version 1 got 0
I20240501 08:12:25.723861 27616 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:25.731117 32339 tablet.cc:1722] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:25.731619 32339 tablet.cc:1620] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11: MemRowSet was empty: no flush needed.
I20240501 08:12:25.732376 32343 tablet.cc:1722] T eaadc7787c3f439b9457f8092df745b3 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:25.732900 32343 tablet.cc:1620] T eaadc7787c3f439b9457f8092df745b3 P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:25.734345 32340 tablet.cc:1722] T eaadc7787c3f439b9457f8092df745b3 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:25.734963 32340 tablet.cc:1620] T eaadc7787c3f439b9457f8092df745b3 P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:25.742045 27616 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=5ac50e5b3c3443c5b510ee241fbdf469] alter complete (version 2)
I20240501 08:12:25.845240 27620 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:25.852434 32339 tablet.cc:1722] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:25.853999 32340 tablet.cc:1722] T eaadc7787c3f439b9457f8092df745b3 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:25.858525 32343 tablet.cc:1722] T eaadc7787c3f439b9457f8092df745b3 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:25.868690 27620 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table_new_name [id=5ac50e5b3c3443c5b510ee241fbdf469] alter complete (version 3)
I20240501 08:12:25.878010 32309 consensus_queue.cc:1035] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:25.884272 32309 consensus_queue.cc:1035] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:25.894539 32344 consensus_queue.cc:1035] T 73058afba1214e21b655283c0612a0ec P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:25.902837 32214 consensus_queue.cc:1035] T 73058afba1214e21b655283c0612a0ec P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:25.972007 27620 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:25.972263 27620 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:25.975427 27620 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet eaadc7787c3f439b9457f8092df745b3
I20240501 08:12:25.976351 30915 tablet_service.cc:1514] Processing DeleteTablet for tablet eaadc7787c3f439b9457f8092df745b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:25 UTC) from {username='slave'} at 127.0.0.1:54036
I20240501 08:12:25.977041 32351 tablet_replica.cc:330] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11: stopping tablet replica
I20240501 08:12:25.977422 32351 raft_consensus.cc:2229] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:25.977703 30517 tablet_service.cc:1514] Processing DeleteTablet for tablet eaadc7787c3f439b9457f8092df745b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:25 UTC) from {username='slave'} at 127.0.0.1:45078
I20240501 08:12:25.977865 32351 raft_consensus.cc:2258] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:25.978937 32351 ts_tablet_manager.cc:1875] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:25.979010 30652 tablet_service.cc:1514] Processing DeleteTablet for tablet eaadc7787c3f439b9457f8092df745b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:25 UTC) from {username='slave'} at 127.0.0.1:57156
I20240501 08:12:25.979758 32353 tablet_replica.cc:330] T eaadc7787c3f439b9457f8092df745b3 P 9a022e1a501f4d44a0468054d6c3de47: stopping tablet replica
I20240501 08:12:25.980000 32353 raft_consensus.cc:2229] T eaadc7787c3f439b9457f8092df745b3 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:25.980233 32353 raft_consensus.cc:2258] T eaadc7787c3f439b9457f8092df745b3 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:25.981223 32353 ts_tablet_manager.cc:1875] T eaadc7787c3f439b9457f8092df745b3 P 9a022e1a501f4d44a0468054d6c3de47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:25.985870 32351 ts_tablet_manager.cc:1888] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:25.986093 32351 log.cc:1196] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/wals/eaadc7787c3f439b9457f8092df745b3
I20240501 08:12:25.986513 32351 ts_tablet_manager.cc:1909] T eaadc7787c3f439b9457f8092df745b3 P a29cd590999b4f32849ff5689cd53c11: Deleting consensus metadata
I20240501 08:12:25.987432 27606 catalog_manager.cc:4830] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): tablet eaadc7787c3f439b9457f8092df745b3 (table rebalancer_dynamic_table_new_name [id=5ac50e5b3c3443c5b510ee241fbdf469]) successfully deleted
I20240501 08:12:25.989171 32352 tablet_replica.cc:330] T eaadc7787c3f439b9457f8092df745b3 P 31811708c6774037b82df44024b229b6: stopping tablet replica
I20240501 08:12:25.989517 32352 raft_consensus.cc:2229] T eaadc7787c3f439b9457f8092df745b3 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:25.989784 32352 raft_consensus.cc:2258] T eaadc7787c3f439b9457f8092df745b3 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:25.990705 32352 ts_tablet_manager.cc:1875] T eaadc7787c3f439b9457f8092df745b3 P 31811708c6774037b82df44024b229b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:25.991140 32353 ts_tablet_manager.cc:1888] T eaadc7787c3f439b9457f8092df745b3 P 9a022e1a501f4d44a0468054d6c3de47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:25.991286 32353 log.cc:1196] T eaadc7787c3f439b9457f8092df745b3 P 9a022e1a501f4d44a0468054d6c3de47: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/wals/eaadc7787c3f439b9457f8092df745b3
I20240501 08:12:25.991797 32353 ts_tablet_manager.cc:1909] T eaadc7787c3f439b9457f8092df745b3 P 9a022e1a501f4d44a0468054d6c3de47: Deleting consensus metadata
I20240501 08:12:25.992841 27604 catalog_manager.cc:4830] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): tablet eaadc7787c3f439b9457f8092df745b3 (table rebalancer_dynamic_table_new_name [id=5ac50e5b3c3443c5b510ee241fbdf469]) successfully deleted
I20240501 08:12:25.995754 32352 ts_tablet_manager.cc:1888] T eaadc7787c3f439b9457f8092df745b3 P 31811708c6774037b82df44024b229b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:25.995870 32352 log.cc:1196] T eaadc7787c3f439b9457f8092df745b3 P 31811708c6774037b82df44024b229b6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/wals/eaadc7787c3f439b9457f8092df745b3
I20240501 08:12:25.996359 32352 ts_tablet_manager.cc:1909] T eaadc7787c3f439b9457f8092df745b3 P 31811708c6774037b82df44024b229b6: Deleting consensus metadata
I20240501 08:12:25.998410 27607 catalog_manager.cc:4830] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): tablet eaadc7787c3f439b9457f8092df745b3 (table rebalancer_dynamic_table_new_name [id=5ac50e5b3c3443c5b510ee241fbdf469]) successfully deleted
I20240501 08:12:26.012357 27620 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_extra_table_46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:26.022163 30784 tablet_service.cc:1467] Processing CreateTablet for tablet 6c324b1136ce429290d7286399bb7121 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=ec67e56c6aaa43f3b2e573534276d8bc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:26.022585 30915 tablet_service.cc:1467] Processing CreateTablet for tablet 6c324b1136ce429290d7286399bb7121 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=ec67e56c6aaa43f3b2e573534276d8bc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:26.022715 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c324b1136ce429290d7286399bb7121. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:26.022984 30915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c324b1136ce429290d7286399bb7121. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:26.025089 27741 tablet_service.cc:1467] Processing CreateTablet for tablet b16391636d5f4ec48c9e4f18ed40badd (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=ec67e56c6aaa43f3b2e573534276d8bc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:26.025533 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b16391636d5f4ec48c9e4f18ed40badd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:26.025502 27871 tablet_service.cc:1467] Processing CreateTablet for tablet 647dba7e192140f9972724bd7e4a2569 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=ec67e56c6aaa43f3b2e573534276d8bc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:26.025584 30652 tablet_service.cc:1467] Processing CreateTablet for tablet 6c324b1136ce429290d7286399bb7121 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=ec67e56c6aaa43f3b2e573534276d8bc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:26.025964 27871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 647dba7e192140f9972724bd7e4a2569. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:26.029146 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c324b1136ce429290d7286399bb7121. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:26.029534 30517 tablet_service.cc:1467] Processing CreateTablet for tablet b16391636d5f4ec48c9e4f18ed40badd (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=ec67e56c6aaa43f3b2e573534276d8bc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:26.030005 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b16391636d5f4ec48c9e4f18ed40badd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:26.033773 30650 tablet_service.cc:1467] Processing CreateTablet for tablet 647dba7e192140f9972724bd7e4a2569 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=ec67e56c6aaa43f3b2e573534276d8bc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:26.034425 30650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 647dba7e192140f9972724bd7e4a2569. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:26.040799 32356 tablet_bootstrap.cc:492] T b16391636d5f4ec48c9e4f18ed40badd P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:26.043228 32356 tablet_bootstrap.cc:654] T b16391636d5f4ec48c9e4f18ed40badd P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:26.044453 30782 tablet_service.cc:1467] Processing CreateTablet for tablet b16391636d5f4ec48c9e4f18ed40badd (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=ec67e56c6aaa43f3b2e573534276d8bc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:26.044613 30781 tablet_service.cc:1467] Processing CreateTablet for tablet 647dba7e192140f9972724bd7e4a2569 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=ec67e56c6aaa43f3b2e573534276d8bc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:26.044907 30782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b16391636d5f4ec48c9e4f18ed40badd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:26.046125 30781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 647dba7e192140f9972724bd7e4a2569. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:26.055678 32359 tablet_bootstrap.cc:492] T b16391636d5f4ec48c9e4f18ed40badd P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:26.058872 32359 tablet_bootstrap.cc:654] T b16391636d5f4ec48c9e4f18ed40badd P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:26.073798 32363 tablet_bootstrap.cc:492] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:26.075690 32363 tablet_bootstrap.cc:654] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:26.079553 32364 tablet_bootstrap.cc:492] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:26.082715 32364 tablet_bootstrap.cc:654] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:26.083690 27616 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:26.084002 32360 tablet_bootstrap.cc:492] T 6c324b1136ce429290d7286399bb7121 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:26.086022 32363 tablet_bootstrap.cc:492] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:26.086876 32363 ts_tablet_manager.cc:1367] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20240501 08:12:26.087908 32363 raft_consensus.cc:348] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:26.088255 32360 tablet_bootstrap.cc:654] T 6c324b1136ce429290d7286399bb7121 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:26.089324 32363 raft_consensus.cc:374] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:26.090152 32363 raft_consensus.cc:729] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:26.090796 32363 consensus_queue.cc:260] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:26.093262 32363 ts_tablet_manager.cc:1398] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:12:26.097512 30517 tablet_service.cc:1467] Processing CreateTablet for tablet d687c73913ef4a968d9f4ac8cd7dee0a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4e68db524e6b4d37a8b7ee0801bc5b21]), partition=
I20240501 08:12:26.097998 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d687c73913ef4a968d9f4ac8cd7dee0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:26.099833 30781 tablet_service.cc:1467] Processing CreateTablet for tablet d687c73913ef4a968d9f4ac8cd7dee0a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4e68db524e6b4d37a8b7ee0801bc5b21]), partition=
I20240501 08:12:26.100224 30781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d687c73913ef4a968d9f4ac8cd7dee0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:26.120735 32357 tablet_bootstrap.cc:492] T 647dba7e192140f9972724bd7e4a2569 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:26.122987 32357 tablet_bootstrap.cc:654] T 647dba7e192140f9972724bd7e4a2569 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:26.125727 32360 tablet_bootstrap.cc:492] T 6c324b1136ce429290d7286399bb7121 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:26.125902 32360 ts_tablet_manager.cc:1367] T 6c324b1136ce429290d7286399bb7121 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.000s
I20240501 08:12:26.126673 32357 tablet_bootstrap.cc:492] T 647dba7e192140f9972724bd7e4a2569 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:12:26.126895 32357 ts_tablet_manager.cc:1367] T 647dba7e192140f9972724bd7e4a2569 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 08:12:26.126868 32360 raft_consensus.cc:348] T 6c324b1136ce429290d7286399bb7121 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:26.127151 32360 raft_consensus.cc:374] T 6c324b1136ce429290d7286399bb7121 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:26.127267 32360 raft_consensus.cc:729] T 6c324b1136ce429290d7286399bb7121 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:26.127537 32360 consensus_queue.cc:260] T 6c324b1136ce429290d7286399bb7121 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:26.129583 32360 ts_tablet_manager.cc:1398] T 6c324b1136ce429290d7286399bb7121 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:26.129875 32360 tablet_bootstrap.cc:492] T 647dba7e192140f9972724bd7e4a2569 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:26.130479 32357 raft_consensus.cc:348] T 647dba7e192140f9972724bd7e4a2569 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:26.130733 32357 raft_consensus.cc:374] T 647dba7e192140f9972724bd7e4a2569 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:26.130827 32357 raft_consensus.cc:729] T 647dba7e192140f9972724bd7e4a2569 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:12:26.131055 32357 consensus_queue.cc:260] T 647dba7e192140f9972724bd7e4a2569 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:26.131629 32357 ts_tablet_manager.cc:1398] T 647dba7e192140f9972724bd7e4a2569 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.004s user 0.000s sys 0.000s
I20240501 08:12:26.131915 32360 tablet_bootstrap.cc:654] T 647dba7e192140f9972724bd7e4a2569 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
W20240501 08:12:26.160154 28087 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.131:36305, user_credentials={real_user=slave}} blocked reactor thread for 114437us
I20240501 08:12:26.167520 32360 tablet_bootstrap.cc:492] T 647dba7e192140f9972724bd7e4a2569 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:26.167775 32360 ts_tablet_manager.cc:1367] T 647dba7e192140f9972724bd7e4a2569 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.038s user 0.015s sys 0.017s
I20240501 08:12:26.167770 28132 tablet_service.cc:1467] Processing CreateTablet for tablet d687c73913ef4a968d9f4ac8cd7dee0a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4e68db524e6b4d37a8b7ee0801bc5b21]), partition=
I20240501 08:12:26.168217 28132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d687c73913ef4a968d9f4ac8cd7dee0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:26.168473 32360 raft_consensus.cc:348] T 647dba7e192140f9972724bd7e4a2569 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:26.168665 32360 raft_consensus.cc:374] T 647dba7e192140f9972724bd7e4a2569 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:26.168740 32360 raft_consensus.cc:729] T 647dba7e192140f9972724bd7e4a2569 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:26.168970 32360 consensus_queue.cc:260] T 647dba7e192140f9972724bd7e4a2569 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:26.169598 32360 ts_tablet_manager.cc:1398] T 647dba7e192140f9972724bd7e4a2569 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:26.180176 32364 tablet_bootstrap.cc:492] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:26.180496 32364 ts_tablet_manager.cc:1367] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.101s user 0.005s sys 0.022s
I20240501 08:12:26.181293 32356 tablet_bootstrap.cc:492] T b16391636d5f4ec48c9e4f18ed40badd P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:12:26.181475 32356 ts_tablet_manager.cc:1367] T b16391636d5f4ec48c9e4f18ed40badd P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.141s user 0.011s sys 0.030s
I20240501 08:12:26.181366 32364 raft_consensus.cc:348] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:26.181728 32364 raft_consensus.cc:374] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:26.181890 32364 raft_consensus.cc:729] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:26.182191 32364 consensus_queue.cc:260] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:26.182304 32356 raft_consensus.cc:348] T b16391636d5f4ec48c9e4f18ed40badd P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:26.182528 32356 raft_consensus.cc:374] T b16391636d5f4ec48c9e4f18ed40badd P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:26.182641 32356 raft_consensus.cc:729] T b16391636d5f4ec48c9e4f18ed40badd P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:12:26.182911 32364 ts_tablet_manager.cc:1398] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:26.182893 32356 consensus_queue.cc:260] T b16391636d5f4ec48c9e4f18ed40badd P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:26.183543 32356 ts_tablet_manager.cc:1398] T b16391636d5f4ec48c9e4f18ed40badd P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:26.184031 32359 tablet_bootstrap.cc:492] T b16391636d5f4ec48c9e4f18ed40badd P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:26.184234 32359 ts_tablet_manager.cc:1367] T b16391636d5f4ec48c9e4f18ed40badd P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.129s user 0.012s sys 0.028s
I20240501 08:12:26.185056 32359 raft_consensus.cc:348] T b16391636d5f4ec48c9e4f18ed40badd P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:26.185282 32359 raft_consensus.cc:374] T b16391636d5f4ec48c9e4f18ed40badd P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:26.185438 32359 raft_consensus.cc:729] T b16391636d5f4ec48c9e4f18ed40badd P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:26.185675 32359 consensus_queue.cc:260] T b16391636d5f4ec48c9e4f18ed40badd P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:26.186316 32359 ts_tablet_manager.cc:1398] T b16391636d5f4ec48c9e4f18ed40badd P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:12:26.186632 32359 tablet_bootstrap.cc:492] T 647dba7e192140f9972724bd7e4a2569 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:26.186923 32364 tablet_bootstrap.cc:492] T d687c73913ef4a968d9f4ac8cd7dee0a P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:26.188603 32359 tablet_bootstrap.cc:654] T 647dba7e192140f9972724bd7e4a2569 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:26.190275 32364 tablet_bootstrap.cc:654] T d687c73913ef4a968d9f4ac8cd7dee0a P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:26.193921 32364 tablet_bootstrap.cc:492] T d687c73913ef4a968d9f4ac8cd7dee0a P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:26.194170 32364 ts_tablet_manager.cc:1367] T d687c73913ef4a968d9f4ac8cd7dee0a P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 08:12:26.194829 32364 raft_consensus.cc:348] T d687c73913ef4a968d9f4ac8cd7dee0a P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:26.195134 32364 raft_consensus.cc:374] T d687c73913ef4a968d9f4ac8cd7dee0a P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:26.195256 32364 raft_consensus.cc:729] T d687c73913ef4a968d9f4ac8cd7dee0a P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:26.195534 32364 consensus_queue.cc:260] T d687c73913ef4a968d9f4ac8cd7dee0a P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:26.196138 32364 ts_tablet_manager.cc:1398] T d687c73913ef4a968d9f4ac8cd7dee0a P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:26.197108 32359 tablet_bootstrap.cc:492] T 647dba7e192140f9972724bd7e4a2569 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:26.197275 32359 ts_tablet_manager.cc:1367] T 647dba7e192140f9972724bd7e4a2569 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20240501 08:12:26.197924 32359 raft_consensus.cc:348] T 647dba7e192140f9972724bd7e4a2569 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:26.198253 32359 raft_consensus.cc:374] T 647dba7e192140f9972724bd7e4a2569 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:26.198380 32359 raft_consensus.cc:729] T 647dba7e192140f9972724bd7e4a2569 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:26.198705 32359 consensus_queue.cc:260] T 647dba7e192140f9972724bd7e4a2569 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:26.199330 32359 ts_tablet_manager.cc:1398] T 647dba7e192140f9972724bd7e4a2569 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:26.199672 32359 tablet_bootstrap.cc:492] T 6c324b1136ce429290d7286399bb7121 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:26.201334 32359 tablet_bootstrap.cc:654] T 6c324b1136ce429290d7286399bb7121 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:26.206820 32374 tablet_bootstrap.cc:492] T d687c73913ef4a968d9f4ac8cd7dee0a P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:26.208521 32374 tablet_bootstrap.cc:654] T d687c73913ef4a968d9f4ac8cd7dee0a P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:26.210229 32359 tablet_bootstrap.cc:492] T 6c324b1136ce429290d7286399bb7121 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:26.210444 32359 ts_tablet_manager.cc:1367] T 6c324b1136ce429290d7286399bb7121 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.004s
I20240501 08:12:26.211200 32359 raft_consensus.cc:348] T 6c324b1136ce429290d7286399bb7121 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:26.211480 32359 raft_consensus.cc:374] T 6c324b1136ce429290d7286399bb7121 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:26.211628 32359 raft_consensus.cc:729] T 6c324b1136ce429290d7286399bb7121 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:26.211833 32359 consensus_queue.cc:260] T 6c324b1136ce429290d7286399bb7121 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:26.212366 32359 ts_tablet_manager.cc:1398] T 6c324b1136ce429290d7286399bb7121 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:26.212657 32359 tablet_bootstrap.cc:492] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:26.214135 32359 tablet_bootstrap.cc:654] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:26.219887 32359 tablet_bootstrap.cc:492] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:26.220329 32359 ts_tablet_manager.cc:1367] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20240501 08:12:26.221302 32359 raft_consensus.cc:348] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:26.221635 32359 raft_consensus.cc:374] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:26.221752 32359 raft_consensus.cc:729] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:26.222110 32359 consensus_queue.cc:260] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:26.222967 32359 ts_tablet_manager.cc:1398] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:26.223649 32374 tablet_bootstrap.cc:492] T d687c73913ef4a968d9f4ac8cd7dee0a P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:12:26.223855 32374 ts_tablet_manager.cc:1367] T d687c73913ef4a968d9f4ac8cd7dee0a P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240501 08:12:26.226238 32374 raft_consensus.cc:348] T d687c73913ef4a968d9f4ac8cd7dee0a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:26.226410 32374 raft_consensus.cc:374] T d687c73913ef4a968d9f4ac8cd7dee0a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:26.226478 32374 raft_consensus.cc:729] T d687c73913ef4a968d9f4ac8cd7dee0a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:12:26.227800 32374 consensus_queue.cc:260] T d687c73913ef4a968d9f4ac8cd7dee0a P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:26.229491 32374 ts_tablet_manager.cc:1398] T d687c73913ef4a968d9f4ac8cd7dee0a P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:26.362810 32303 raft_consensus.cc:482] T 647dba7e192140f9972724bd7e4a2569 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:26.363193 32303 raft_consensus.cc:504] T 647dba7e192140f9972724bd7e4a2569 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:26.364109 32303 leader_election.cc:288] T 647dba7e192140f9972724bd7e4a2569 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:12:26.365739 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647dba7e192140f9972724bd7e4a2569" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:12:26.366068 27889 raft_consensus.cc:2454] T 647dba7e192140f9972724bd7e4a2569 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:26.366945 30740 leader_election.cc:302] T 647dba7e192140f9972724bd7e4a2569 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:26.368008 32378 raft_consensus.cc:2781] T 647dba7e192140f9972724bd7e4a2569 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:26.368129 32378 raft_consensus.cc:482] T 647dba7e192140f9972724bd7e4a2569 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:26.368230 32378 raft_consensus.cc:3037] T 647dba7e192140f9972724bd7e4a2569 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:26.368710 30669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647dba7e192140f9972724bd7e4a2569" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:26.368805 32214 raft_consensus.cc:482] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:26.368971 30669 raft_consensus.cc:2454] T 647dba7e192140f9972724bd7e4a2569 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:26.368981 32214 raft_consensus.cc:504] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:26.369747 32214 leader_election.cc:288] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:26.370976 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c324b1136ce429290d7286399bb7121" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:26.373342 32378 raft_consensus.cc:504] T 647dba7e192140f9972724bd7e4a2569 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:26.374150 32378 leader_election.cc:288] T 647dba7e192140f9972724bd7e4a2569 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Requested vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:12:26.374693 30669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647dba7e192140f9972724bd7e4a2569" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:26.374903 30669 raft_consensus.cc:3037] T 647dba7e192140f9972724bd7e4a2569 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:26.375270 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647dba7e192140f9972724bd7e4a2569" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:12:26.375519 27889 raft_consensus.cc:3037] T 647dba7e192140f9972724bd7e4a2569 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:26.377460 30804 raft_consensus.cc:2454] T 6c324b1136ce429290d7286399bb7121 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:26.378304 30669 raft_consensus.cc:2454] T 647dba7e192140f9972724bd7e4a2569 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:26.378994 30737 leader_election.cc:302] T 647dba7e192140f9972724bd7e4a2569 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:26.379323 32378 raft_consensus.cc:2781] T 647dba7e192140f9972724bd7e4a2569 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:26.379616 32378 raft_consensus.cc:686] T 647dba7e192140f9972724bd7e4a2569 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Becoming Leader. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Running, Role: LEADER
I20240501 08:12:26.380162 32378 consensus_queue.cc:237] T 647dba7e192140f9972724bd7e4a2569 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:26.381129 30871 leader_election.cc:302] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:26.384442 32214 raft_consensus.cc:2781] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:26.384581 32214 raft_consensus.cc:482] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:26.384660 32214 raft_consensus.cc:3037] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:26.387131 32214 raft_consensus.cc:504] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:26.388453 32214 leader_election.cc:288] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Requested vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:26.389807 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c324b1136ce429290d7286399bb7121" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:26.390331 30804 raft_consensus.cc:3037] T 6c324b1136ce429290d7286399bb7121 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:26.399245 27613 catalog_manager.cc:5484] T 647dba7e192140f9972724bd7e4a2569 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64998323a1f4482d8704bf20594d13a5 (127.21.84.135). New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:26.405587 27889 raft_consensus.cc:2454] T 647dba7e192140f9972724bd7e4a2569 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:26.406526 30669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c324b1136ce429290d7286399bb7121" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:26.406810 30669 raft_consensus.cc:2454] T 6c324b1136ce429290d7286399bb7121 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:26.407192 30670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c324b1136ce429290d7286399bb7121" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:26.407379 30670 raft_consensus.cc:3037] T 6c324b1136ce429290d7286399bb7121 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:26.408180 30804 raft_consensus.cc:2454] T 6c324b1136ce429290d7286399bb7121 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:26.410370 30670 raft_consensus.cc:2454] T 6c324b1136ce429290d7286399bb7121 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:26.410755 30870 leader_election.cc:302] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a022e1a501f4d44a0468054d6c3de47, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:26.411032 32344 raft_consensus.cc:2781] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:26.411805 32344 raft_consensus.cc:686] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Becoming Leader. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:12:26.412144 32344 consensus_queue.cc:237] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:26.417028 27613 catalog_manager.cc:5484] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: term changed from 0 to 1, leader changed from <none> to a29cd590999b4f32849ff5689cd53c11 (127.21.84.136). New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:26.454479 32325 raft_consensus.cc:482] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:26.454653 32325 raft_consensus.cc:504] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:26.454983 32378 raft_consensus.cc:482] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:26.455147 32378 raft_consensus.cc:504] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:26.455847 32378 leader_election.cc:288] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:26.456243 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d687c73913ef4a968d9f4ac8cd7dee0a" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:12:26.456275 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16391636d5f4ec48c9e4f18ed40badd" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:26.456472 28150 raft_consensus.cc:2454] T d687c73913ef4a968d9f4ac8cd7dee0a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:26.456614 30804 raft_consensus.cc:2454] T b16391636d5f4ec48c9e4f18ed40badd P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 0.
I20240501 08:12:26.456923 30737 leader_election.cc:302] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:12:26.457019 30471 leader_election.cc:302] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 64998323a1f4482d8704bf20594d13a5; no voters:
I20240501 08:12:26.457355 32378 raft_consensus.cc:2781] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:26.455586 32325 leader_election.cc:288] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:26.457536 32354 raft_consensus.cc:2781] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:26.457542 32378 raft_consensus.cc:482] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:26.457652 32354 raft_consensus.cc:482] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:26.457765 32354 raft_consensus.cc:3037] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:26.457794 32378 raft_consensus.cc:3037] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:26.460672 32354 raft_consensus.cc:504] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:26.462200 32354 leader_election.cc:288] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 election: Requested vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:26.464917 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d687c73913ef4a968d9f4ac8cd7dee0a" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:26.465180 30537 raft_consensus.cc:2454] T d687c73913ef4a968d9f4ac8cd7dee0a P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:26.467723 32378 raft_consensus.cc:504] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:26.468642 32378 leader_election.cc:288] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Requested vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:26.469844 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d687c73913ef4a968d9f4ac8cd7dee0a" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:12:26.470117 28150 raft_consensus.cc:3037] T d687c73913ef4a968d9f4ac8cd7dee0a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:26.472282 28150 raft_consensus.cc:2454] T d687c73913ef4a968d9f4ac8cd7dee0a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:26.472805 30737 leader_election.cc:302] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:12:26.473140 32378 raft_consensus.cc:2781] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:26.473091 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d687c73913ef4a968d9f4ac8cd7dee0a" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:26.473325 30537 raft_consensus.cc:3037] T d687c73913ef4a968d9f4ac8cd7dee0a P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:26.473629 32378 raft_consensus.cc:686] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Becoming Leader. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Running, Role: LEADER
I20240501 08:12:26.473992 32378 consensus_queue.cc:237] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:26.477437 30537 raft_consensus.cc:2454] T d687c73913ef4a968d9f4ac8cd7dee0a P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:26.478199 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16391636d5f4ec48c9e4f18ed40badd" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:26.478432 30804 raft_consensus.cc:3037] T b16391636d5f4ec48c9e4f18ed40badd P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:26.479849 27613 catalog_manager.cc:5484] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64998323a1f4482d8704bf20594d13a5 (127.21.84.135). New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:26.480342 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16391636d5f4ec48c9e4f18ed40badd" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:12:26.480580 27759 raft_consensus.cc:2454] T b16391636d5f4ec48c9e4f18ed40badd P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 0.
I20240501 08:12:26.480410 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b16391636d5f4ec48c9e4f18ed40badd" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:12:26.481057 27761 raft_consensus.cc:3037] T b16391636d5f4ec48c9e4f18ed40badd P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:26.485044 27761 raft_consensus.cc:2454] T b16391636d5f4ec48c9e4f18ed40badd P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 1.
I20240501 08:12:26.485288 30804 raft_consensus.cc:2454] T b16391636d5f4ec48c9e4f18ed40badd P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 1.
I20240501 08:12:26.486022 30472 leader_election.cc:302] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 6331f77dae9d452184e8cd0cbf5f1b43; no voters:
I20240501 08:12:26.486394 32325 raft_consensus.cc:2781] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:26.486547 32325 raft_consensus.cc:686] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Becoming Leader. State: Replica: 31811708c6774037b82df44024b229b6, State: Running, Role: LEADER
I20240501 08:12:26.486820 32325 consensus_queue.cc:237] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:26.491560 27613 catalog_manager.cc:5484] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31811708c6774037b82df44024b229b6 (127.21.84.133). New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:26.627689 27613 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:26.647781 30536 raft_consensus.cc:1261] T d687c73913ef4a968d9f4ac8cd7dee0a P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:26.648387 28150 raft_consensus.cc:1261] T d687c73913ef4a968d9f4ac8cd7dee0a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:26.648447 32378 consensus_queue.cc:1035] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:26.649042 32302 consensus_queue.cc:1035] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:26.653430 32388 tablet.cc:1722] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:26.654377 32388 tablet.cc:1620] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5: MemRowSet was empty: no flush needed.
I20240501 08:12:26.655045 27618 catalog_manager.cc:5576] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293) does not have the latest schema for tablet d687c73913ef4a968d9f4ac8cd7dee0a (table rebalancer_dynamic_table [id=4e68db524e6b4d37a8b7ee0801bc5b21]). Expected version 1 got 0
I20240501 08:12:26.660710 27613 catalog_manager.cc:5576] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323) does not have the latest schema for tablet d687c73913ef4a968d9f4ac8cd7dee0a (table rebalancer_dynamic_table [id=4e68db524e6b4d37a8b7ee0801bc5b21]). Expected version 1 got 0
I20240501 08:12:26.661968 32389 tablet.cc:1722] T d687c73913ef4a968d9f4ac8cd7dee0a P 91d11eae5ef6432ca86eb8ed446b6ea6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:26.662820 32389 tablet.cc:1620] T d687c73913ef4a968d9f4ac8cd7dee0a P 91d11eae5ef6432ca86eb8ed446b6ea6: MemRowSet was empty: no flush needed.
I20240501 08:12:26.662777 27620 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=4e68db524e6b4d37a8b7ee0801bc5b21] alter complete (version 1)
I20240501 08:12:26.665591 27613 catalog_manager.cc:5576] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293) does not have the latest schema for tablet d687c73913ef4a968d9f4ac8cd7dee0a (table rebalancer_dynamic_table [id=4e68db524e6b4d37a8b7ee0801bc5b21]). Expected version 1 got 0
I20240501 08:12:26.670002 32390 tablet.cc:1722] T d687c73913ef4a968d9f4ac8cd7dee0a P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:26.670554 32390 tablet.cc:1620] T d687c73913ef4a968d9f4ac8cd7dee0a P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:26.769203 27613 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:26.777205 32388 tablet.cc:1722] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:26.777737 32388 tablet.cc:1620] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5: MemRowSet was empty: no flush needed.
I20240501 08:12:26.777667 32389 tablet.cc:1722] T d687c73913ef4a968d9f4ac8cd7dee0a P 91d11eae5ef6432ca86eb8ed446b6ea6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:26.778151 32389 tablet.cc:1620] T d687c73913ef4a968d9f4ac8cd7dee0a P 91d11eae5ef6432ca86eb8ed446b6ea6: MemRowSet was empty: no flush needed.
I20240501 08:12:26.782224 32390 tablet.cc:1722] T d687c73913ef4a968d9f4ac8cd7dee0a P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:26.782666 32390 tablet.cc:1620] T d687c73913ef4a968d9f4ac8cd7dee0a P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:26.787979 27613 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=4e68db524e6b4d37a8b7ee0801bc5b21] alter complete (version 2)
I20240501 08:12:26.825255 32302 consensus_queue.cc:1035] T 647dba7e192140f9972724bd7e4a2569 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:26.829219 32302 consensus_queue.cc:1035] T 647dba7e192140f9972724bd7e4a2569 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:26.863436 32309 consensus_queue.cc:1035] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:26.865703 32325 consensus_queue.cc:1035] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:26.871616 32354 consensus_queue.cc:1035] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:26.882639 32309 consensus_queue.cc:1035] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:26.892452 27613 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:26.901525 32388 tablet.cc:1722] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:26.902166 32390 tablet.cc:1722] T d687c73913ef4a968d9f4ac8cd7dee0a P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:26.905265 32389 tablet.cc:1722] T d687c73913ef4a968d9f4ac8cd7dee0a P 91d11eae5ef6432ca86eb8ed446b6ea6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:26.915608 27620 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table_new_name [id=4e68db524e6b4d37a8b7ee0801bc5b21] alter complete (version 3)
I20240501 08:12:27.024966 27620 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:27.025259 27620 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:27.027734 27618 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_extra_table_48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:27.028424 27620 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet d687c73913ef4a968d9f4ac8cd7dee0a
I20240501 08:12:27.029228 30517 tablet_service.cc:1514] Processing DeleteTablet for tablet d687c73913ef4a968d9f4ac8cd7dee0a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:27 UTC) from {username='slave'} at 127.0.0.1:45078
I20240501 08:12:27.029273 28132 tablet_service.cc:1514] Processing DeleteTablet for tablet d687c73913ef4a968d9f4ac8cd7dee0a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:27 UTC) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:27.029680 30781 tablet_service.cc:1514] Processing DeleteTablet for tablet d687c73913ef4a968d9f4ac8cd7dee0a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:27 UTC) from {username='slave'} at 127.0.0.1:38952
I20240501 08:12:27.031280 32404 tablet_replica.cc:330] T d687c73913ef4a968d9f4ac8cd7dee0a P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:27.031567 32404 raft_consensus.cc:2229] T d687c73913ef4a968d9f4ac8cd7dee0a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:27.031822 32405 tablet_replica.cc:330] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5: stopping tablet replica
I20240501 08:12:27.031836 32404 raft_consensus.cc:2258] T d687c73913ef4a968d9f4ac8cd7dee0a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:27.032049 32403 tablet_replica.cc:330] T d687c73913ef4a968d9f4ac8cd7dee0a P 31811708c6774037b82df44024b229b6: stopping tablet replica
I20240501 08:12:27.032130 32405 raft_consensus.cc:2229] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:27.032343 32403 raft_consensus.cc:2229] T d687c73913ef4a968d9f4ac8cd7dee0a P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:27.032567 32405 raft_consensus.cc:2258] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:27.032905 32404 ts_tablet_manager.cc:1875] T d687c73913ef4a968d9f4ac8cd7dee0a P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:27.032966 32403 raft_consensus.cc:2258] T d687c73913ef4a968d9f4ac8cd7dee0a P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:27.033639 32405 ts_tablet_manager.cc:1875] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:27.033979 32403 ts_tablet_manager.cc:1875] T d687c73913ef4a968d9f4ac8cd7dee0a P 31811708c6774037b82df44024b229b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:27.039443 32404 ts_tablet_manager.cc:1888] T d687c73913ef4a968d9f4ac8cd7dee0a P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:27.039671 32404 log.cc:1196] T d687c73913ef4a968d9f4ac8cd7dee0a P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/d687c73913ef4a968d9f4ac8cd7dee0a
I20240501 08:12:27.040213 32404 ts_tablet_manager.cc:1909] T d687c73913ef4a968d9f4ac8cd7dee0a P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting consensus metadata
I20240501 08:12:27.040719 27741 tablet_service.cc:1467] Processing CreateTablet for tablet 019d8108d6d84d608fd8d5237f3f20ff (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=e75e40faa06f4299a600ba7063fa7b4b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:27.041287 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 019d8108d6d84d608fd8d5237f3f20ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:27.041618 30514 tablet_service.cc:1467] Processing CreateTablet for tablet 019d8108d6d84d608fd8d5237f3f20ff (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=e75e40faa06f4299a600ba7063fa7b4b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:27.042093 30514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 019d8108d6d84d608fd8d5237f3f20ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:27.043470 30915 tablet_service.cc:1467] Processing CreateTablet for tablet 019d8108d6d84d608fd8d5237f3f20ff (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=e75e40faa06f4299a600ba7063fa7b4b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:27.043943 30915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 019d8108d6d84d608fd8d5237f3f20ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:27.046442 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet d687c73913ef4a968d9f4ac8cd7dee0a (table rebalancer_dynamic_table_new_name [id=4e68db524e6b4d37a8b7ee0801bc5b21]) successfully deleted
I20240501 08:12:27.050873 32405 ts_tablet_manager.cc:1888] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:27.051155 32405 log.cc:1196] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal/wals/d687c73913ef4a968d9f4ac8cd7dee0a
I20240501 08:12:27.052649 32403 ts_tablet_manager.cc:1888] T d687c73913ef4a968d9f4ac8cd7dee0a P 31811708c6774037b82df44024b229b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:27.052800 32403 log.cc:1196] T d687c73913ef4a968d9f4ac8cd7dee0a P 31811708c6774037b82df44024b229b6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/wals/d687c73913ef4a968d9f4ac8cd7dee0a
I20240501 08:12:27.056643 32403 ts_tablet_manager.cc:1909] T d687c73913ef4a968d9f4ac8cd7dee0a P 31811708c6774037b82df44024b229b6: Deleting consensus metadata
I20240501 08:12:27.056847 32405 ts_tablet_manager.cc:1909] T d687c73913ef4a968d9f4ac8cd7dee0a P 64998323a1f4482d8704bf20594d13a5: Deleting consensus metadata
I20240501 08:12:27.058019 27605 catalog_manager.cc:4830] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): tablet d687c73913ef4a968d9f4ac8cd7dee0a (table rebalancer_dynamic_table_new_name [id=4e68db524e6b4d37a8b7ee0801bc5b21]) successfully deleted
I20240501 08:12:27.058661 27739 tablet_service.cc:1467] Processing CreateTablet for tablet 9e1b0cfacf5148a5998c356f1c6f006f (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=e75e40faa06f4299a600ba7063fa7b4b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:27.059139 27739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e1b0cfacf5148a5998c356f1c6f006f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:27.061132 27871 tablet_service.cc:1467] Processing CreateTablet for tablet 9e1b0cfacf5148a5998c356f1c6f006f (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=e75e40faa06f4299a600ba7063fa7b4b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:27.061638 27871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e1b0cfacf5148a5998c356f1c6f006f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:27.066701 32406 tablet_bootstrap.cc:492] T 019d8108d6d84d608fd8d5237f3f20ff P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:27.067270 32408 tablet_bootstrap.cc:492] T 019d8108d6d84d608fd8d5237f3f20ff P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:27.068099 27607 catalog_manager.cc:4830] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): tablet d687c73913ef4a968d9f4ac8cd7dee0a (table rebalancer_dynamic_table_new_name [id=4e68db524e6b4d37a8b7ee0801bc5b21]) successfully deleted
I20240501 08:12:27.069862 32408 tablet_bootstrap.cc:654] T 019d8108d6d84d608fd8d5237f3f20ff P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:27.071767 32406 tablet_bootstrap.cc:654] T 019d8108d6d84d608fd8d5237f3f20ff P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:27.072311 27872 tablet_service.cc:1467] Processing CreateTablet for tablet a1fde6b765374df9bc4674a5d678c920 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=e75e40faa06f4299a600ba7063fa7b4b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:27.072907 27872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1fde6b765374df9bc4674a5d678c920. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:27.075552 32408 tablet_bootstrap.cc:492] T 019d8108d6d84d608fd8d5237f3f20ff P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:12:27.075696 32408 ts_tablet_manager.cc:1367] T 019d8108d6d84d608fd8d5237f3f20ff P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240501 08:12:27.079115 32408 raft_consensus.cc:348] T 019d8108d6d84d608fd8d5237f3f20ff P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:27.080106 32408 raft_consensus.cc:374] T 019d8108d6d84d608fd8d5237f3f20ff P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:27.080466 32408 raft_consensus.cc:729] T 019d8108d6d84d608fd8d5237f3f20ff P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:12:27.080897 32408 consensus_queue.cc:260] T 019d8108d6d84d608fd8d5237f3f20ff P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:27.082532 32408 ts_tablet_manager.cc:1398] T 019d8108d6d84d608fd8d5237f3f20ff P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240501 08:12:27.083102 32408 tablet_bootstrap.cc:492] T 9e1b0cfacf5148a5998c356f1c6f006f P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:27.085165 32408 tablet_bootstrap.cc:654] T 9e1b0cfacf5148a5998c356f1c6f006f P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:27.093916 32406 tablet_bootstrap.cc:492] T 019d8108d6d84d608fd8d5237f3f20ff P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:27.094005 32408 tablet_bootstrap.cc:492] T 9e1b0cfacf5148a5998c356f1c6f006f P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:12:27.094180 32408 ts_tablet_manager.cc:1367] T 9e1b0cfacf5148a5998c356f1c6f006f P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240501 08:12:27.095033 32408 raft_consensus.cc:348] T 9e1b0cfacf5148a5998c356f1c6f006f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:27.095328 32408 raft_consensus.cc:374] T 9e1b0cfacf5148a5998c356f1c6f006f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:27.095430 32408 raft_consensus.cc:729] T 9e1b0cfacf5148a5998c356f1c6f006f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:12:27.095768 32408 consensus_queue.cc:260] T 9e1b0cfacf5148a5998c356f1c6f006f P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:27.096351 32408 ts_tablet_manager.cc:1398] T 9e1b0cfacf5148a5998c356f1c6f006f P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:27.094290 32406 ts_tablet_manager.cc:1367] T 019d8108d6d84d608fd8d5237f3f20ff P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.006s
I20240501 08:12:27.097815 32406 raft_consensus.cc:348] T 019d8108d6d84d608fd8d5237f3f20ff P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:27.098271 32406 raft_consensus.cc:374] T 019d8108d6d84d608fd8d5237f3f20ff P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:27.098443 32406 raft_consensus.cc:729] T 019d8108d6d84d608fd8d5237f3f20ff P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:27.098799 32406 consensus_queue.cc:260] T 019d8108d6d84d608fd8d5237f3f20ff P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:27.101280 32406 ts_tablet_manager.cc:1398] T 019d8108d6d84d608fd8d5237f3f20ff P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:27.102130 32371 raft_consensus.cc:482] T 019d8108d6d84d608fd8d5237f3f20ff P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:27.102315 32371 raft_consensus.cc:504] T 019d8108d6d84d608fd8d5237f3f20ff P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:27.103487 32371 leader_election.cc:288] T 019d8108d6d84d608fd8d5237f3f20ff P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:27.105814 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019d8108d6d84d608fd8d5237f3f20ff" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:27.106110 30536 raft_consensus.cc:2454] T 019d8108d6d84d608fd8d5237f3f20ff P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:12:27.106688 27697 leader_election.cc:302] T 019d8108d6d84d608fd8d5237f3f20ff P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 6331f77dae9d452184e8cd0cbf5f1b43; no voters:
I20240501 08:12:27.107796 32346 raft_consensus.cc:2781] T 019d8108d6d84d608fd8d5237f3f20ff P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:27.107935 32346 raft_consensus.cc:482] T 019d8108d6d84d608fd8d5237f3f20ff P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:27.108103 32346 raft_consensus.cc:3037] T 019d8108d6d84d608fd8d5237f3f20ff P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:27.108572 30781 tablet_service.cc:1467] Processing CreateTablet for tablet a1fde6b765374df9bc4674a5d678c920 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=e75e40faa06f4299a600ba7063fa7b4b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:27.109010 30781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1fde6b765374df9bc4674a5d678c920. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:27.110069 28002 tablet_service.cc:1467] Processing CreateTablet for tablet a1fde6b765374df9bc4674a5d678c920 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=e75e40faa06f4299a600ba7063fa7b4b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:27.110500 28002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1fde6b765374df9bc4674a5d678c920. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:27.111894 32346 raft_consensus.cc:504] T 019d8108d6d84d608fd8d5237f3f20ff P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:27.113565 32346 leader_election.cc:288] T 019d8108d6d84d608fd8d5237f3f20ff P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Requested vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:27.127704 32412 tablet_bootstrap.cc:492] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:27.129531 32415 tablet_bootstrap.cc:492] T 9e1b0cfacf5148a5998c356f1c6f006f P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:27.130682 32412 tablet_bootstrap.cc:654] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:27.132040 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019d8108d6d84d608fd8d5237f3f20ff" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:27.132318 30536 raft_consensus.cc:3037] T 019d8108d6d84d608fd8d5237f3f20ff P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:27.132460 27620 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:27.135177 30536 raft_consensus.cc:2454] T 019d8108d6d84d608fd8d5237f3f20ff P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:12:27.136607 30652 tablet_service.cc:1467] Processing CreateTablet for tablet 9e1b0cfacf5148a5998c356f1c6f006f (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=e75e40faa06f4299a600ba7063fa7b4b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:27.137069 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e1b0cfacf5148a5998c356f1c6f006f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:27.137291 27697 leader_election.cc:302] T 019d8108d6d84d608fd8d5237f3f20ff P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 6331f77dae9d452184e8cd0cbf5f1b43; no voters:
I20240501 08:12:27.137542 32412 tablet_bootstrap.cc:492] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:27.137553 32415 tablet_bootstrap.cc:654] T 9e1b0cfacf5148a5998c356f1c6f006f P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:27.137684 32412 ts_tablet_manager.cc:1367] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240501 08:12:27.137697 32346 raft_consensus.cc:2781] T 019d8108d6d84d608fd8d5237f3f20ff P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:27.138572 32412 raft_consensus.cc:348] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:27.138842 32412 raft_consensus.cc:374] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:27.138942 32412 raft_consensus.cc:729] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:27.141736 32412 consensus_queue.cc:260] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
W20240501 08:12:27.144905 27696 leader_election.cc:341] T 019d8108d6d84d608fd8d5237f3f20ff P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): Not found: Tablet not found: 019d8108d6d84d608fd8d5237f3f20ff
I20240501 08:12:27.145316 30935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019d8108d6d84d608fd8d5237f3f20ff" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:27.146319 32412 ts_tablet_manager.cc:1398] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
W20240501 08:12:27.146428 27696 leader_election.cc:341] T 019d8108d6d84d608fd8d5237f3f20ff P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:27.148284 32415 tablet_bootstrap.cc:492] T 9e1b0cfacf5148a5998c356f1c6f006f P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:12:27.149288 32415 ts_tablet_manager.cc:1367] T 9e1b0cfacf5148a5998c356f1c6f006f P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.004s
I20240501 08:12:27.153224 32415 raft_consensus.cc:348] T 9e1b0cfacf5148a5998c356f1c6f006f P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:27.154086 32415 raft_consensus.cc:374] T 9e1b0cfacf5148a5998c356f1c6f006f P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:27.154389 32415 raft_consensus.cc:729] T 9e1b0cfacf5148a5998c356f1c6f006f P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:12:27.154870 32415 consensus_queue.cc:260] T 9e1b0cfacf5148a5998c356f1c6f006f P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:27.156124 32415 ts_tablet_manager.cc:1398] T 9e1b0cfacf5148a5998c356f1c6f006f P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 08:12:27.156474 32415 tablet_bootstrap.cc:492] T a1fde6b765374df9bc4674a5d678c920 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:27.158298 32415 tablet_bootstrap.cc:654] T a1fde6b765374df9bc4674a5d678c920 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:27.182219 30781 tablet_service.cc:1467] Processing CreateTablet for tablet 47d0970dd7404fa6a7788617c00e4d04 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a1037661c4f44800bcf6ffc8039fde24]), partition=
I20240501 08:12:27.182641 30781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47d0970dd7404fa6a7788617c00e4d04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:27.188809 30916 tablet_service.cc:1467] Processing CreateTablet for tablet 47d0970dd7404fa6a7788617c00e4d04 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a1037661c4f44800bcf6ffc8039fde24]), partition=
I20240501 08:12:27.189172 30916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47d0970dd7404fa6a7788617c00e4d04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:27.190557 32412 tablet_bootstrap.cc:492] T 47d0970dd7404fa6a7788617c00e4d04 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:27.192129 32412 tablet_bootstrap.cc:654] T 47d0970dd7404fa6a7788617c00e4d04 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:27.199898 30650 tablet_service.cc:1467] Processing CreateTablet for tablet 47d0970dd7404fa6a7788617c00e4d04 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a1037661c4f44800bcf6ffc8039fde24]), partition=
I20240501 08:12:27.200414 30650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47d0970dd7404fa6a7788617c00e4d04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:27.208618 32412 tablet_bootstrap.cc:492] T 47d0970dd7404fa6a7788617c00e4d04 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:27.208941 32412 ts_tablet_manager.cc:1367] T 47d0970dd7404fa6a7788617c00e4d04 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.007s
I20240501 08:12:27.210862 32412 raft_consensus.cc:348] T 47d0970dd7404fa6a7788617c00e4d04 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:27.211267 32412 raft_consensus.cc:374] T 47d0970dd7404fa6a7788617c00e4d04 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:27.211397 32412 raft_consensus.cc:729] T 47d0970dd7404fa6a7788617c00e4d04 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:27.211766 32412 consensus_queue.cc:260] T 47d0970dd7404fa6a7788617c00e4d04 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:27.214254 32412 ts_tablet_manager.cc:1398] T 47d0970dd7404fa6a7788617c00e4d04 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:27.219415 32421 tablet_bootstrap.cc:492] T 9e1b0cfacf5148a5998c356f1c6f006f P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:27.222136 32421 tablet_bootstrap.cc:654] T 9e1b0cfacf5148a5998c356f1c6f006f P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:27.225322 32423 tablet_bootstrap.cc:492] T a1fde6b765374df9bc4674a5d678c920 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:27.232187 32346 raft_consensus.cc:686] T 019d8108d6d84d608fd8d5237f3f20ff P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:12:27.232813 32346 consensus_queue.cc:237] T 019d8108d6d84d608fd8d5237f3f20ff P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:27.235240 32413 tablet_bootstrap.cc:492] T 019d8108d6d84d608fd8d5237f3f20ff P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:27.242980 32423 tablet_bootstrap.cc:654] T a1fde6b765374df9bc4674a5d678c920 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:27.244973 32413 tablet_bootstrap.cc:654] T 019d8108d6d84d608fd8d5237f3f20ff P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:27.248644 27620 catalog_manager.cc:5484] T 019d8108d6d84d608fd8d5237f3f20ff P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:27.259465 32423 tablet_bootstrap.cc:492] T a1fde6b765374df9bc4674a5d678c920 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:12:27.259624 32423 ts_tablet_manager.cc:1367] T a1fde6b765374df9bc4674a5d678c920 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.000s
I20240501 08:12:27.261963 32421 tablet_bootstrap.cc:492] T 9e1b0cfacf5148a5998c356f1c6f006f P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:27.262158 32421 ts_tablet_manager.cc:1367] T 9e1b0cfacf5148a5998c356f1c6f006f P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.043s user 0.002s sys 0.003s
I20240501 08:12:27.262956 32421 raft_consensus.cc:348] T 9e1b0cfacf5148a5998c356f1c6f006f P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:27.263167 32421 raft_consensus.cc:374] T 9e1b0cfacf5148a5998c356f1c6f006f P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:27.263239 32421 raft_consensus.cc:729] T 9e1b0cfacf5148a5998c356f1c6f006f P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:27.264097 32423 raft_consensus.cc:348] T a1fde6b765374df9bc4674a5d678c920 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:27.264369 32423 raft_consensus.cc:374] T a1fde6b765374df9bc4674a5d678c920 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:27.264469 32423 raft_consensus.cc:729] T a1fde6b765374df9bc4674a5d678c920 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:12:27.265002 32423 consensus_queue.cc:260] T a1fde6b765374df9bc4674a5d678c920 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:27.265718 32423 ts_tablet_manager.cc:1398] T a1fde6b765374df9bc4674a5d678c920 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240501 08:12:27.266938 32413 tablet_bootstrap.cc:492] T 019d8108d6d84d608fd8d5237f3f20ff P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:27.268215 32413 ts_tablet_manager.cc:1367] T 019d8108d6d84d608fd8d5237f3f20ff P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.000s
I20240501 08:12:27.269134 32413 raft_consensus.cc:348] T 019d8108d6d84d608fd8d5237f3f20ff P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:27.269454 32413 raft_consensus.cc:374] T 019d8108d6d84d608fd8d5237f3f20ff P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:27.269608 32413 raft_consensus.cc:729] T 019d8108d6d84d608fd8d5237f3f20ff P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:27.269902 32413 consensus_queue.cc:260] T 019d8108d6d84d608fd8d5237f3f20ff P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:27.270743 32413 ts_tablet_manager.cc:1398] T 019d8108d6d84d608fd8d5237f3f20ff P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:27.271070 32413 tablet_bootstrap.cc:492] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:27.272147 32421 consensus_queue.cc:260] T 9e1b0cfacf5148a5998c356f1c6f006f P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:27.272846 32421 ts_tablet_manager.cc:1398] T 9e1b0cfacf5148a5998c356f1c6f006f P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240501 08:12:27.272950 32413 tablet_bootstrap.cc:654] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:27.274930 32415 tablet_bootstrap.cc:492] T a1fde6b765374df9bc4674a5d678c920 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:12:27.275405 32415 ts_tablet_manager.cc:1367] T a1fde6b765374df9bc4674a5d678c920 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.119s user 0.036s sys 0.043s
I20240501 08:12:27.276298 32415 raft_consensus.cc:348] T a1fde6b765374df9bc4674a5d678c920 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:27.276526 32415 raft_consensus.cc:374] T a1fde6b765374df9bc4674a5d678c920 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:27.276593 32415 raft_consensus.cc:729] T a1fde6b765374df9bc4674a5d678c920 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:12:27.276858 32415 consensus_queue.cc:260] T a1fde6b765374df9bc4674a5d678c920 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:27.277554 32415 ts_tablet_manager.cc:1398] T a1fde6b765374df9bc4674a5d678c920 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:27.278187 32421 tablet_bootstrap.cc:492] T 47d0970dd7404fa6a7788617c00e4d04 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:27.280172 32421 tablet_bootstrap.cc:654] T 47d0970dd7404fa6a7788617c00e4d04 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:27.290673 32421 tablet_bootstrap.cc:492] T 47d0970dd7404fa6a7788617c00e4d04 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:27.290961 32421 ts_tablet_manager.cc:1367] T 47d0970dd7404fa6a7788617c00e4d04 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.010s
I20240501 08:12:27.291880 32421 raft_consensus.cc:348] T 47d0970dd7404fa6a7788617c00e4d04 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:27.292197 32421 raft_consensus.cc:374] T 47d0970dd7404fa6a7788617c00e4d04 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:27.292317 32421 raft_consensus.cc:729] T 47d0970dd7404fa6a7788617c00e4d04 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:27.292616 32421 consensus_queue.cc:260] T 47d0970dd7404fa6a7788617c00e4d04 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:27.293213 32421 ts_tablet_manager.cc:1398] T 47d0970dd7404fa6a7788617c00e4d04 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:12:27.307199 32413 tablet_bootstrap.cc:492] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:27.307380 32413 ts_tablet_manager.cc:1367] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.036s user 0.006s sys 0.021s
I20240501 08:12:27.308188 32413 raft_consensus.cc:348] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:27.308462 32413 raft_consensus.cc:374] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:27.308617 32413 raft_consensus.cc:729] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:27.308864 32413 consensus_queue.cc:260] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:27.309538 32413 ts_tablet_manager.cc:1398] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:27.311136 32214 raft_consensus.cc:482] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:27.311300 32214 raft_consensus.cc:504] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:27.312040 32214 leader_election.cc:288] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:27.312443 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d0970dd7404fa6a7788617c00e4d04" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:27.312659 30804 raft_consensus.cc:2454] T 47d0970dd7404fa6a7788617c00e4d04 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:27.312753 30671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d0970dd7404fa6a7788617c00e4d04" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:27.313001 30671 raft_consensus.cc:2454] T 47d0970dd7404fa6a7788617c00e4d04 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:27.313261 30871 leader_election.cc:302] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:27.314345 32214 raft_consensus.cc:2781] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:27.314473 32214 raft_consensus.cc:482] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:27.314576 32214 raft_consensus.cc:3037] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:27.317376 32214 raft_consensus.cc:504] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:27.318763 30671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d0970dd7404fa6a7788617c00e4d04" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:27.318805 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d0970dd7404fa6a7788617c00e4d04" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:27.318979 30671 raft_consensus.cc:3037] T 47d0970dd7404fa6a7788617c00e4d04 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:27.319015 30804 raft_consensus.cc:3037] T 47d0970dd7404fa6a7788617c00e4d04 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:27.321633 30804 raft_consensus.cc:2454] T 47d0970dd7404fa6a7788617c00e4d04 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:27.321988 30671 raft_consensus.cc:2454] T 47d0970dd7404fa6a7788617c00e4d04 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:27.322160 30871 leader_election.cc:302] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:27.322494 32344 raft_consensus.cc:2781] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:27.322635 32344 raft_consensus.cc:686] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Becoming Leader. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:12:27.322713 32214 leader_election.cc:288] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Requested vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:27.322930 32344 consensus_queue.cc:237] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:27.331353 27620 catalog_manager.cc:5484] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: term changed from 0 to 1, leader changed from <none> to a29cd590999b4f32849ff5689cd53c11 (127.21.84.136). New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:27.407832 32417 raft_consensus.cc:482] T 9e1b0cfacf5148a5998c356f1c6f006f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:27.407991 32417 raft_consensus.cc:504] T 9e1b0cfacf5148a5998c356f1c6f006f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:27.408913 32417 leader_election.cc:288] T 9e1b0cfacf5148a5998c356f1c6f006f P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:12:27.409376 30671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e1b0cfacf5148a5998c356f1c6f006f" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:27.409422 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e1b0cfacf5148a5998c356f1c6f006f" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:12:27.409565 30671 raft_consensus.cc:2454] T 9e1b0cfacf5148a5998c356f1c6f006f P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:12:27.409727 27890 raft_consensus.cc:2454] T 9e1b0cfacf5148a5998c356f1c6f006f P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:12:27.410035 27694 leader_election.cc:302] T 9e1b0cfacf5148a5998c356f1c6f006f P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:27.410362 32417 raft_consensus.cc:2781] T 9e1b0cfacf5148a5998c356f1c6f006f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:27.410492 32417 raft_consensus.cc:482] T 9e1b0cfacf5148a5998c356f1c6f006f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:27.410596 32417 raft_consensus.cc:3037] T 9e1b0cfacf5148a5998c356f1c6f006f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:27.413175 32417 raft_consensus.cc:504] T 9e1b0cfacf5148a5998c356f1c6f006f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:27.413754 32417 leader_election.cc:288] T 9e1b0cfacf5148a5998c356f1c6f006f P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Requested vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:12:27.414368 30671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e1b0cfacf5148a5998c356f1c6f006f" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:27.414551 30671 raft_consensus.cc:3037] T 9e1b0cfacf5148a5998c356f1c6f006f P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:27.414538 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e1b0cfacf5148a5998c356f1c6f006f" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:12:27.414764 27890 raft_consensus.cc:3037] T 9e1b0cfacf5148a5998c356f1c6f006f P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:27.416501 30671 raft_consensus.cc:2454] T 9e1b0cfacf5148a5998c356f1c6f006f P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:12:27.416886 27890 raft_consensus.cc:2454] T 9e1b0cfacf5148a5998c356f1c6f006f P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:12:27.416952 27694 leader_election.cc:302] T 9e1b0cfacf5148a5998c356f1c6f006f P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:27.417321 32417 raft_consensus.cc:2781] T 9e1b0cfacf5148a5998c356f1c6f006f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:27.417486 32417 raft_consensus.cc:686] T 9e1b0cfacf5148a5998c356f1c6f006f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:12:27.417762 32417 consensus_queue.cc:237] T 9e1b0cfacf5148a5998c356f1c6f006f P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:27.421413 27620 catalog_manager.cc:5484] T 9e1b0cfacf5148a5998c356f1c6f006f P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:27.446204 27620 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:27.454324 30670 raft_consensus.cc:1261] T 47d0970dd7404fa6a7788617c00e4d04 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:27.454483 30804 raft_consensus.cc:1261] T 47d0970dd7404fa6a7788617c00e4d04 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:27.454943 32344 consensus_queue.cc:1035] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:27.455238 32309 consensus_queue.cc:1035] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:27.479166 27620 catalog_manager.cc:5576] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933) does not have the latest schema for tablet 47d0970dd7404fa6a7788617c00e4d04 (table rebalancer_dynamic_table [id=a1037661c4f44800bcf6ffc8039fde24]). Expected version 1 got 0
I20240501 08:12:27.481463 32439 tablet.cc:1722] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:27.483680 32439 tablet.cc:1620] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11: MemRowSet was empty: no flush needed.
I20240501 08:12:27.485109 27616 catalog_manager.cc:5576] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503) does not have the latest schema for tablet 47d0970dd7404fa6a7788617c00e4d04 (table rebalancer_dynamic_table [id=a1037661c4f44800bcf6ffc8039fde24]). Expected version 1 got 0
I20240501 08:12:27.488816 32435 mvcc.cc:205] Tried to move back new op lower bound from 7022801499966111744 to 7022801499437015040. Current Snapshot: MvccSnapshot[applied={T|T < 7022801499966111744}]
I20240501 08:12:27.492215 27620 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=a1037661c4f44800bcf6ffc8039fde24] alter complete (version 1)
I20240501 08:12:27.495857 32440 tablet.cc:1722] T 47d0970dd7404fa6a7788617c00e4d04 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:27.499620 32441 tablet.cc:1722] T 47d0970dd7404fa6a7788617c00e4d04 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:27.501250 27616 catalog_manager.cc:5576] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503) does not have the latest schema for tablet 47d0970dd7404fa6a7788617c00e4d04 (table rebalancer_dynamic_table [id=a1037661c4f44800bcf6ffc8039fde24]). Expected version 1 got 0
I20240501 08:12:27.503509 27613 catalog_manager.cc:5576] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933) does not have the latest schema for tablet 47d0970dd7404fa6a7788617c00e4d04 (table rebalancer_dynamic_table [id=a1037661c4f44800bcf6ffc8039fde24]). Expected version 1 got 0
I20240501 08:12:27.507068 32441 tablet.cc:1620] T 47d0970dd7404fa6a7788617c00e4d04 P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:27.496467 32440 tablet.cc:1620] T 47d0970dd7404fa6a7788617c00e4d04 P 64998323a1f4482d8704bf20594d13a5: MemRowSet was empty: no flush needed.
I20240501 08:12:27.558529 32302 raft_consensus.cc:482] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:27.558744 32302 raft_consensus.cc:504] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:27.559501 32302 leader_election.cc:288] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:27.559967 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1fde6b765374df9bc4674a5d678c920" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:12:27.560237 27890 raft_consensus.cc:2454] T a1fde6b765374df9bc4674a5d678c920 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:27.560426 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1fde6b765374df9bc4674a5d678c920" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:12:27.560659 28022 raft_consensus.cc:2454] T a1fde6b765374df9bc4674a5d678c920 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:27.562247 30739 leader_election.cc:302] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:12:27.562577 32302 raft_consensus.cc:2781] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:27.562674 32302 raft_consensus.cc:482] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:27.562778 32302 raft_consensus.cc:3037] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:27.565659 32302 raft_consensus.cc:504] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:27.566505 32302 leader_election.cc:288] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Requested vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:27.567113 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1fde6b765374df9bc4674a5d678c920" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:12:27.567345 27889 raft_consensus.cc:3037] T a1fde6b765374df9bc4674a5d678c920 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:27.567479 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1fde6b765374df9bc4674a5d678c920" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:12:27.567670 28021 raft_consensus.cc:3037] T a1fde6b765374df9bc4674a5d678c920 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:27.571394 28021 raft_consensus.cc:2454] T a1fde6b765374df9bc4674a5d678c920 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:27.572398 30739 leader_election.cc:302] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:12:27.572772 32302 raft_consensus.cc:2781] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:27.572983 32302 raft_consensus.cc:686] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Becoming Leader. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Running, Role: LEADER
I20240501 08:12:27.573390 32302 consensus_queue.cc:237] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:27.575883 27889 raft_consensus.cc:2454] T a1fde6b765374df9bc4674a5d678c920 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:27.581068 27620 catalog_manager.cc:5484] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64998323a1f4482d8704bf20594d13a5 (127.21.84.135). New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:27.598599 27620 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:27.606087 32439 tablet.cc:1722] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:27.606631 32439 tablet.cc:1620] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11: MemRowSet was empty: no flush needed.
I20240501 08:12:27.607349 32441 tablet.cc:1722] T 47d0970dd7404fa6a7788617c00e4d04 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:27.607837 32441 tablet.cc:1620] T 47d0970dd7404fa6a7788617c00e4d04 P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:27.608268 32440 tablet.cc:1722] T 47d0970dd7404fa6a7788617c00e4d04 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:27.608736 32440 tablet.cc:1620] T 47d0970dd7404fa6a7788617c00e4d04 P 64998323a1f4482d8704bf20594d13a5: MemRowSet was empty: no flush needed.
I20240501 08:12:27.618387 27620 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=a1037661c4f44800bcf6ffc8039fde24] alter complete (version 2)
I20240501 08:12:27.636145 32214 raft_consensus.cc:482] T 019d8108d6d84d608fd8d5237f3f20ff P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:27.636317 32214 raft_consensus.cc:504] T 019d8108d6d84d608fd8d5237f3f20ff P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:27.636935 32214 leader_election.cc:288] T 019d8108d6d84d608fd8d5237f3f20ff P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:27.637413 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019d8108d6d84d608fd8d5237f3f20ff" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:27.637580 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019d8108d6d84d608fd8d5237f3f20ff" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:12:27.637712 30536 raft_consensus.cc:2379] T 019d8108d6d84d608fd8d5237f3f20ff P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a29cd590999b4f32849ff5689cd53c11 in current term 1: Already voted for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in this term.
I20240501 08:12:27.638240 30872 leader_election.cc:302] T 019d8108d6d84d608fd8d5237f3f20ff P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a29cd590999b4f32849ff5689cd53c11; no voters: 31811708c6774037b82df44024b229b6, 6331f77dae9d452184e8cd0cbf5f1b43
I20240501 08:12:27.638499 32214 raft_consensus.cc:3037] T 019d8108d6d84d608fd8d5237f3f20ff P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:27.640717 32214 raft_consensus.cc:2726] T 019d8108d6d84d608fd8d5237f3f20ff P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:27.650786 32417 consensus_queue.cc:1035] T 019d8108d6d84d608fd8d5237f3f20ff P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:27.655627 32417 consensus_queue.cc:1035] T 019d8108d6d84d608fd8d5237f3f20ff P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:27.720676 27618 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:27.728780 32439 tablet.cc:1722] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:27.730311 32441 tablet.cc:1722] T 47d0970dd7404fa6a7788617c00e4d04 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:27.731458 32440 tablet.cc:1722] T 47d0970dd7404fa6a7788617c00e4d04 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:27.739719 27618 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table_new_name [id=a1037661c4f44800bcf6ffc8039fde24] alter complete (version 3)
I20240501 08:12:27.845966 27620 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:27.846222 27620 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:27.849258 27620 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 47d0970dd7404fa6a7788617c00e4d04
I20240501 08:12:27.850214 30781 tablet_service.cc:1514] Processing DeleteTablet for tablet 47d0970dd7404fa6a7788617c00e4d04 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:27 UTC) from {username='slave'} at 127.0.0.1:38952
I20240501 08:12:27.850268 30915 tablet_service.cc:1514] Processing DeleteTablet for tablet 47d0970dd7404fa6a7788617c00e4d04 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:27 UTC) from {username='slave'} at 127.0.0.1:54036
I20240501 08:12:27.850946 30650 tablet_service.cc:1514] Processing DeleteTablet for tablet 47d0970dd7404fa6a7788617c00e4d04 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:27 UTC) from {username='slave'} at 127.0.0.1:57156
I20240501 08:12:27.851210 32453 tablet_replica.cc:330] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11: stopping tablet replica
I20240501 08:12:27.851418 32453 raft_consensus.cc:2229] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:27.851519 32454 tablet_replica.cc:330] T 47d0970dd7404fa6a7788617c00e4d04 P 9a022e1a501f4d44a0468054d6c3de47: stopping tablet replica
I20240501 08:12:27.851766 32453 raft_consensus.cc:2258] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:27.851801 32454 raft_consensus.cc:2229] T 47d0970dd7404fa6a7788617c00e4d04 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:27.852115 32454 raft_consensus.cc:2258] T 47d0970dd7404fa6a7788617c00e4d04 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:27.852636 32453 ts_tablet_manager.cc:1875] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:27.853132 32454 ts_tablet_manager.cc:1875] T 47d0970dd7404fa6a7788617c00e4d04 P 9a022e1a501f4d44a0468054d6c3de47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:27.857278 32453 ts_tablet_manager.cc:1888] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:27.857465 32453 log.cc:1196] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/wals/47d0970dd7404fa6a7788617c00e4d04
I20240501 08:12:27.858016 32453 ts_tablet_manager.cc:1909] T 47d0970dd7404fa6a7788617c00e4d04 P a29cd590999b4f32849ff5689cd53c11: Deleting consensus metadata
I20240501 08:12:27.859413 27606 catalog_manager.cc:4830] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): tablet 47d0970dd7404fa6a7788617c00e4d04 (table rebalancer_dynamic_table_new_name [id=a1037661c4f44800bcf6ffc8039fde24]) successfully deleted
I20240501 08:12:27.859606 32452 tablet_replica.cc:330] T 47d0970dd7404fa6a7788617c00e4d04 P 64998323a1f4482d8704bf20594d13a5: stopping tablet replica
I20240501 08:12:27.859829 32452 raft_consensus.cc:2229] T 47d0970dd7404fa6a7788617c00e4d04 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:27.860015 32452 raft_consensus.cc:2258] T 47d0970dd7404fa6a7788617c00e4d04 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:27.860622 32452 ts_tablet_manager.cc:1875] T 47d0970dd7404fa6a7788617c00e4d04 P 64998323a1f4482d8704bf20594d13a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:27.861011 32454 ts_tablet_manager.cc:1888] T 47d0970dd7404fa6a7788617c00e4d04 P 9a022e1a501f4d44a0468054d6c3de47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:27.861166 32454 log.cc:1196] T 47d0970dd7404fa6a7788617c00e4d04 P 9a022e1a501f4d44a0468054d6c3de47: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/wals/47d0970dd7404fa6a7788617c00e4d04
I20240501 08:12:27.861616 32454 ts_tablet_manager.cc:1909] T 47d0970dd7404fa6a7788617c00e4d04 P 9a022e1a501f4d44a0468054d6c3de47: Deleting consensus metadata
I20240501 08:12:27.862510 32417 consensus_queue.cc:1035] T 9e1b0cfacf5148a5998c356f1c6f006f P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:27.863121 27604 catalog_manager.cc:4830] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): tablet 47d0970dd7404fa6a7788617c00e4d04 (table rebalancer_dynamic_table_new_name [id=a1037661c4f44800bcf6ffc8039fde24]) successfully deleted
I20240501 08:12:27.867533 32346 consensus_queue.cc:1035] T 9e1b0cfacf5148a5998c356f1c6f006f P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:27.869213 32452 ts_tablet_manager.cc:1888] T 47d0970dd7404fa6a7788617c00e4d04 P 64998323a1f4482d8704bf20594d13a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:27.869403 32452 log.cc:1196] T 47d0970dd7404fa6a7788617c00e4d04 P 64998323a1f4482d8704bf20594d13a5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal/wals/47d0970dd7404fa6a7788617c00e4d04
I20240501 08:12:27.870249 32452 ts_tablet_manager.cc:1909] T 47d0970dd7404fa6a7788617c00e4d04 P 64998323a1f4482d8704bf20594d13a5: Deleting consensus metadata
I20240501 08:12:27.871438 27605 catalog_manager.cc:4830] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): tablet 47d0970dd7404fa6a7788617c00e4d04 (table rebalancer_dynamic_table_new_name [id=a1037661c4f44800bcf6ffc8039fde24]) successfully deleted
I20240501 08:12:27.952481 27618 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:27.959931 30650 tablet_service.cc:1467] Processing CreateTablet for tablet 94210ebbc379459d8b6e24237d33bb73 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ef4f42d7d291452da72bd7f47df1e35c]), partition=
I20240501 08:12:27.960000 28132 tablet_service.cc:1467] Processing CreateTablet for tablet 94210ebbc379459d8b6e24237d33bb73 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ef4f42d7d291452da72bd7f47df1e35c]), partition=
I20240501 08:12:27.960335 30650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94210ebbc379459d8b6e24237d33bb73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:27.960347 28132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94210ebbc379459d8b6e24237d33bb73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:27.965359 32459 tablet_bootstrap.cc:492] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:27.965617 32460 tablet_bootstrap.cc:492] T 94210ebbc379459d8b6e24237d33bb73 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:27.966246 30781 tablet_service.cc:1467] Processing CreateTablet for tablet 94210ebbc379459d8b6e24237d33bb73 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ef4f42d7d291452da72bd7f47df1e35c]), partition=
I20240501 08:12:27.966655 30781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94210ebbc379459d8b6e24237d33bb73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:27.967133 32459 tablet_bootstrap.cc:654] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:27.967456 32460 tablet_bootstrap.cc:654] T 94210ebbc379459d8b6e24237d33bb73 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:27.981581 32464 tablet_bootstrap.cc:492] T 94210ebbc379459d8b6e24237d33bb73 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:27.982156 32459 tablet_bootstrap.cc:492] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:12:27.982717 32459 ts_tablet_manager.cc:1367] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.002s
I20240501 08:12:27.983696 32459 raft_consensus.cc:348] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:27.984089 32459 raft_consensus.cc:374] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:27.984258 32459 raft_consensus.cc:729] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:12:27.984652 32459 consensus_queue.cc:260] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:27.985543 32459 ts_tablet_manager.cc:1398] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:27.986475 32464 tablet_bootstrap.cc:654] T 94210ebbc379459d8b6e24237d33bb73 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:27.987211 32460 tablet_bootstrap.cc:492] T 94210ebbc379459d8b6e24237d33bb73 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:27.987454 32460 ts_tablet_manager.cc:1367] T 94210ebbc379459d8b6e24237d33bb73 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.011s
I20240501 08:12:27.988296 32460 raft_consensus.cc:348] T 94210ebbc379459d8b6e24237d33bb73 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:27.988617 32460 raft_consensus.cc:374] T 94210ebbc379459d8b6e24237d33bb73 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:27.988754 32460 raft_consensus.cc:729] T 94210ebbc379459d8b6e24237d33bb73 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:27.989032 32460 consensus_queue.cc:260] T 94210ebbc379459d8b6e24237d33bb73 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:27.990236 32460 ts_tablet_manager.cc:1398] T 94210ebbc379459d8b6e24237d33bb73 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:12:27.993499 32464 tablet_bootstrap.cc:492] T 94210ebbc379459d8b6e24237d33bb73 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:27.993681 32464 ts_tablet_manager.cc:1367] T 94210ebbc379459d8b6e24237d33bb73 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240501 08:12:27.994426 32464 raft_consensus.cc:348] T 94210ebbc379459d8b6e24237d33bb73 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:27.994621 32464 raft_consensus.cc:374] T 94210ebbc379459d8b6e24237d33bb73 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:27.994715 32464 raft_consensus.cc:729] T 94210ebbc379459d8b6e24237d33bb73 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:27.994936 32464 consensus_queue.cc:260] T 94210ebbc379459d8b6e24237d33bb73 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:27.995466 32464 ts_tablet_manager.cc:1398] T 94210ebbc379459d8b6e24237d33bb73 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:12:28.104149 32378 consensus_queue.cc:1035] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:28.117996 32467 consensus_queue.cc:1035] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:28.131881 27618 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_extra_table_50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:28.141691 30650 tablet_service.cc:1467] Processing CreateTablet for tablet 80bb0e53796f4b319fa9f877451076f3 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=95faa655da1346f68f96519e9154b964]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:28.142062 30517 tablet_service.cc:1467] Processing CreateTablet for tablet 80bb0e53796f4b319fa9f877451076f3 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=95faa655da1346f68f96519e9154b964]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:28.142061 28132 tablet_service.cc:1467] Processing CreateTablet for tablet 80bb0e53796f4b319fa9f877451076f3 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=95faa655da1346f68f96519e9154b964]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:28.142251 30650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80bb0e53796f4b319fa9f877451076f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:28.142475 28132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80bb0e53796f4b319fa9f877451076f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:28.142501 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80bb0e53796f4b319fa9f877451076f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:28.142542 30915 tablet_service.cc:1467] Processing CreateTablet for tablet 6c9bb7ab554e4023892339cd9e0c2a75 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=95faa655da1346f68f96519e9154b964]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:28.142925 30915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c9bb7ab554e4023892339cd9e0c2a75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:28.148468 30652 tablet_service.cc:1467] Processing CreateTablet for tablet 6c9bb7ab554e4023892339cd9e0c2a75 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=95faa655da1346f68f96519e9154b964]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:28.148986 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c9bb7ab554e4023892339cd9e0c2a75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:28.149281 32459 tablet_bootstrap.cc:492] T 80bb0e53796f4b319fa9f877451076f3 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:28.149814 30781 tablet_service.cc:1467] Processing CreateTablet for tablet 6c9bb7ab554e4023892339cd9e0c2a75 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=95faa655da1346f68f96519e9154b964]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:28.150108 32460 tablet_bootstrap.cc:492] T 80bb0e53796f4b319fa9f877451076f3 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:28.150267 30781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c9bb7ab554e4023892339cd9e0c2a75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:28.151068 32459 tablet_bootstrap.cc:654] T 80bb0e53796f4b319fa9f877451076f3 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:28.151710 32460 tablet_bootstrap.cc:654] T 80bb0e53796f4b319fa9f877451076f3 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:28.162371 27741 tablet_service.cc:1467] Processing CreateTablet for tablet 0ac9368b897246ac8b2f6b2f096ff3ea (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=95faa655da1346f68f96519e9154b964]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:28.162899 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ac9368b897246ac8b2f6b2f096ff3ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:28.162905 30650 tablet_service.cc:1467] Processing CreateTablet for tablet 0ac9368b897246ac8b2f6b2f096ff3ea (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=95faa655da1346f68f96519e9154b964]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:28.163098 28002 tablet_service.cc:1467] Processing CreateTablet for tablet 0ac9368b897246ac8b2f6b2f096ff3ea (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=95faa655da1346f68f96519e9154b964]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:28.163302 30650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ac9368b897246ac8b2f6b2f096ff3ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:28.163537 28002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ac9368b897246ac8b2f6b2f096ff3ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:28.170467 32459 tablet_bootstrap.cc:492] T 80bb0e53796f4b319fa9f877451076f3 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:12:28.170773 32459 ts_tablet_manager.cc:1367] T 80bb0e53796f4b319fa9f877451076f3 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.004s
I20240501 08:12:28.172091 32474 tablet_bootstrap.cc:492] T 0ac9368b897246ac8b2f6b2f096ff3ea P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:28.171963 32459 raft_consensus.cc:348] T 80bb0e53796f4b319fa9f877451076f3 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:28.172288 32459 raft_consensus.cc:374] T 80bb0e53796f4b319fa9f877451076f3 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:28.172386 32459 raft_consensus.cc:729] T 80bb0e53796f4b319fa9f877451076f3 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:12:28.172677 32459 consensus_queue.cc:260] T 80bb0e53796f4b319fa9f877451076f3 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:28.174896 32451 raft_consensus.cc:482] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:28.175037 32459 ts_tablet_manager.cc:1398] T 80bb0e53796f4b319fa9f877451076f3 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:28.175567 32472 tablet_bootstrap.cc:492] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:28.175078 32451 raft_consensus.cc:504] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:28.177527 32451 leader_election.cc:288] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:28.177994 32473 tablet_bootstrap.cc:492] T 0ac9368b897246ac8b2f6b2f096ff3ea P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:28.178262 30670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94210ebbc379459d8b6e24237d33bb73" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:28.179183 30670 raft_consensus.cc:2454] T 94210ebbc379459d8b6e24237d33bb73 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:12:28.179693 28085 leader_election.cc:302] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:28.180342 32451 raft_consensus.cc:2781] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:28.180454 32451 raft_consensus.cc:482] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:28.180537 32451 raft_consensus.cc:3037] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:28.180639 32472 tablet_bootstrap.cc:654] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:28.176578 32474 tablet_bootstrap.cc:654] T 0ac9368b897246ac8b2f6b2f096ff3ea P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:28.181962 32460 tablet_bootstrap.cc:492] T 80bb0e53796f4b319fa9f877451076f3 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:28.182122 32460 ts_tablet_manager.cc:1367] T 80bb0e53796f4b319fa9f877451076f3 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.032s user 0.008s sys 0.010s
I20240501 08:12:28.183189 32473 tablet_bootstrap.cc:654] T 0ac9368b897246ac8b2f6b2f096ff3ea P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:28.183501 32451 raft_consensus.cc:504] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:28.184720 32460 raft_consensus.cc:348] T 80bb0e53796f4b319fa9f877451076f3 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:28.185108 32460 raft_consensus.cc:374] T 80bb0e53796f4b319fa9f877451076f3 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:28.185240 32460 raft_consensus.cc:729] T 80bb0e53796f4b319fa9f877451076f3 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:28.185586 32460 consensus_queue.cc:260] T 80bb0e53796f4b319fa9f877451076f3 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:28.186482 32460 ts_tablet_manager.cc:1398] T 80bb0e53796f4b319fa9f877451076f3 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:28.186789 30670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94210ebbc379459d8b6e24237d33bb73" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:28.187021 30670 raft_consensus.cc:3037] T 94210ebbc379459d8b6e24237d33bb73 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:28.187084 32460 tablet_bootstrap.cc:492] T 6c9bb7ab554e4023892339cd9e0c2a75 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:28.194926 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94210ebbc379459d8b6e24237d33bb73" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:28.195199 30804 raft_consensus.cc:2454] T 94210ebbc379459d8b6e24237d33bb73 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:12:28.195336 32460 tablet_bootstrap.cc:654] T 6c9bb7ab554e4023892339cd9e0c2a75 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:28.197500 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94210ebbc379459d8b6e24237d33bb73" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:28.197724 30804 raft_consensus.cc:3037] T 94210ebbc379459d8b6e24237d33bb73 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:28.200469 30804 raft_consensus.cc:2454] T 94210ebbc379459d8b6e24237d33bb73 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:12:28.206230 28086 leader_election.cc:302] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:12:28.208060 30670 raft_consensus.cc:2454] T 94210ebbc379459d8b6e24237d33bb73 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:12:28.208379 32458 raft_consensus.cc:2781] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:28.208583 32458 raft_consensus.cc:686] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:12:28.208689 32472 tablet_bootstrap.cc:492] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:28.208827 32472 ts_tablet_manager.cc:1367] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.047s user 0.012s sys 0.000s
I20240501 08:12:28.208923 32458 consensus_queue.cc:237] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:28.209707 32472 raft_consensus.cc:348] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:28.209904 32472 raft_consensus.cc:374] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:28.210049 32472 raft_consensus.cc:729] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:28.210338 32472 consensus_queue.cc:260] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:28.210620 32443 raft_consensus.cc:482] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:28.210794 32443 raft_consensus.cc:504] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:28.210932 32472 ts_tablet_manager.cc:1398] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:28.211617 32451 leader_election.cc:288] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:28.211779 32443 leader_election.cc:288] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:28.215991 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80bb0e53796f4b319fa9f877451076f3" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:12:28.216254 28151 raft_consensus.cc:2454] T 80bb0e53796f4b319fa9f877451076f3 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 0.
I20240501 08:12:28.216946 30470 leader_election.cc:302] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:12:28.217417 32443 raft_consensus.cc:2781] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:28.217553 32443 raft_consensus.cc:482] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:28.217651 32443 raft_consensus.cc:3037] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:28.219918 32460 tablet_bootstrap.cc:492] T 6c9bb7ab554e4023892339cd9e0c2a75 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:28.220064 32460 ts_tablet_manager.cc:1367] T 6c9bb7ab554e4023892339cd9e0c2a75 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.033s user 0.013s sys 0.007s
I20240501 08:12:28.220587 32443 raft_consensus.cc:504] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:28.221045 32460 raft_consensus.cc:348] T 6c9bb7ab554e4023892339cd9e0c2a75 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:28.222093 32460 raft_consensus.cc:374] T 6c9bb7ab554e4023892339cd9e0c2a75 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:28.222218 32460 raft_consensus.cc:729] T 6c9bb7ab554e4023892339cd9e0c2a75 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:28.222476 32460 consensus_queue.cc:260] T 6c9bb7ab554e4023892339cd9e0c2a75 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:28.223183 32460 ts_tablet_manager.cc:1398] T 6c9bb7ab554e4023892339cd9e0c2a75 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:12:28.223492 32460 tablet_bootstrap.cc:492] T 0ac9368b897246ac8b2f6b2f096ff3ea P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:28.223968 32464 tablet_bootstrap.cc:492] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:28.224000 30671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80bb0e53796f4b319fa9f877451076f3" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:28.224243 30671 raft_consensus.cc:3037] T 80bb0e53796f4b319fa9f877451076f3 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:28.225669 32464 tablet_bootstrap.cc:654] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:28.225996 32460 tablet_bootstrap.cc:654] T 0ac9368b897246ac8b2f6b2f096ff3ea P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:28.227612 30671 raft_consensus.cc:2454] T 80bb0e53796f4b319fa9f877451076f3 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 1.
I20240501 08:12:28.228029 30470 leader_election.cc:302] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:28.228366 32444 raft_consensus.cc:2781] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:28.232062 32443 leader_election.cc:288] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 election: Requested vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:28.233135 30670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80bb0e53796f4b319fa9f877451076f3" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:28.234865 30670 raft_consensus.cc:2362] T 80bb0e53796f4b319fa9f877451076f3 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 31811708c6774037b82df44024b229b6 in term 1. Re-sending same reply.
I20240501 08:12:28.235695 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80bb0e53796f4b319fa9f877451076f3" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:12:28.235926 28151 raft_consensus.cc:3037] T 80bb0e53796f4b319fa9f877451076f3 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:28.240214 32473 tablet_bootstrap.cc:492] T 0ac9368b897246ac8b2f6b2f096ff3ea P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:12:28.240942 32473 ts_tablet_manager.cc:1367] T 0ac9368b897246ac8b2f6b2f096ff3ea P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.063s user 0.005s sys 0.008s
I20240501 08:12:28.241948 28151 raft_consensus.cc:2454] T 80bb0e53796f4b319fa9f877451076f3 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 1.
I20240501 08:12:28.242130 32473 raft_consensus.cc:348] T 0ac9368b897246ac8b2f6b2f096ff3ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:28.242394 32473 raft_consensus.cc:374] T 0ac9368b897246ac8b2f6b2f096ff3ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:28.242497 32473 raft_consensus.cc:729] T 0ac9368b897246ac8b2f6b2f096ff3ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:12:28.243742 32460 tablet_bootstrap.cc:492] T 0ac9368b897246ac8b2f6b2f096ff3ea P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:28.244055 32460 ts_tablet_manager.cc:1367] T 0ac9368b897246ac8b2f6b2f096ff3ea P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.001s
I20240501 08:12:28.245131 32473 consensus_queue.cc:260] T 0ac9368b897246ac8b2f6b2f096ff3ea P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:28.246353 32473 ts_tablet_manager.cc:1398] T 0ac9368b897246ac8b2f6b2f096ff3ea P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:12:28.247546 32460 raft_consensus.cc:348] T 0ac9368b897246ac8b2f6b2f096ff3ea P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:28.247871 32460 raft_consensus.cc:374] T 0ac9368b897246ac8b2f6b2f096ff3ea P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:28.247992 32460 raft_consensus.cc:729] T 0ac9368b897246ac8b2f6b2f096ff3ea P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:28.248235 32444 raft_consensus.cc:686] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Becoming Leader. State: Replica: 31811708c6774037b82df44024b229b6, State: Running, Role: LEADER
I20240501 08:12:28.248353 32460 consensus_queue.cc:260] T 0ac9368b897246ac8b2f6b2f096ff3ea P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:28.248133 27620 catalog_manager.cc:5484] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:28.248618 32444 consensus_queue.cc:237] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:28.250439 32464 tablet_bootstrap.cc:492] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:28.250624 32464 ts_tablet_manager.cc:1367] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.005s
I20240501 08:12:28.251829 32460 ts_tablet_manager.cc:1398] T 0ac9368b897246ac8b2f6b2f096ff3ea P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 08:12:28.252197 32464 raft_consensus.cc:348] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:28.252540 32464 raft_consensus.cc:374] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:28.252686 32464 raft_consensus.cc:729] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:28.252972 32464 consensus_queue.cc:260] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:28.253746 32464 ts_tablet_manager.cc:1398] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240501 08:12:28.268384 32378 raft_consensus.cc:482] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:28.268573 32378 raft_consensus.cc:504] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:28.269281 32378 leader_election.cc:288] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:28.270140 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c9bb7ab554e4023892339cd9e0c2a75" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
W20240501 08:12:28.270875 30739 leader_election.cc:341] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:28.271384 30670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c9bb7ab554e4023892339cd9e0c2a75" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:28.271622 30670 raft_consensus.cc:2454] T 6c9bb7ab554e4023892339cd9e0c2a75 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:28.271952 30737 leader_election.cc:302] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 9a022e1a501f4d44a0468054d6c3de47; no voters: a29cd590999b4f32849ff5689cd53c11
I20240501 08:12:28.272238 32378 raft_consensus.cc:2781] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:28.272385 32378 raft_consensus.cc:482] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:28.272508 32378 raft_consensus.cc:3037] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:28.274302 32475 tablet_bootstrap.cc:492] T 6c9bb7ab554e4023892339cd9e0c2a75 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:28.275933 32378 raft_consensus.cc:504] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:28.277036 32475 tablet_bootstrap.cc:654] T 6c9bb7ab554e4023892339cd9e0c2a75 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:28.277293 30670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c9bb7ab554e4023892339cd9e0c2a75" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:28.277612 30670 raft_consensus.cc:3037] T 6c9bb7ab554e4023892339cd9e0c2a75 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:28.277760 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c9bb7ab554e4023892339cd9e0c2a75" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
W20240501 08:12:28.278507 30739 leader_election.cc:341] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:28.279284 32378 leader_election.cc:288] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Requested vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:28.279361 32475 tablet_bootstrap.cc:492] T 6c9bb7ab554e4023892339cd9e0c2a75 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:28.280172 32474 tablet_bootstrap.cc:492] T 0ac9368b897246ac8b2f6b2f096ff3ea P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:12:28.280387 32474 ts_tablet_manager.cc:1367] T 0ac9368b897246ac8b2f6b2f096ff3ea P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.108s user 0.005s sys 0.010s
I20240501 08:12:28.280251 27618 catalog_manager.cc:5484] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31811708c6774037b82df44024b229b6 (127.21.84.133). New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:28.281215 32474 raft_consensus.cc:348] T 0ac9368b897246ac8b2f6b2f096ff3ea P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:28.281478 32474 raft_consensus.cc:374] T 0ac9368b897246ac8b2f6b2f096ff3ea P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:28.281610 32474 raft_consensus.cc:729] T 0ac9368b897246ac8b2f6b2f096ff3ea P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:12:28.281985 32474 consensus_queue.cc:260] T 0ac9368b897246ac8b2f6b2f096ff3ea P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:28.282800 32474 ts_tablet_manager.cc:1398] T 0ac9368b897246ac8b2f6b2f096ff3ea P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:28.283176 32475 ts_tablet_manager.cc:1367] T 6c9bb7ab554e4023892339cd9e0c2a75 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20240501 08:12:28.284050 32475 raft_consensus.cc:348] T 6c9bb7ab554e4023892339cd9e0c2a75 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:28.284341 32475 raft_consensus.cc:374] T 6c9bb7ab554e4023892339cd9e0c2a75 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:28.284354 30670 raft_consensus.cc:2454] T 6c9bb7ab554e4023892339cd9e0c2a75 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:28.284515 32475 raft_consensus.cc:729] T 6c9bb7ab554e4023892339cd9e0c2a75 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:28.284868 30737 leader_election.cc:302] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 9a022e1a501f4d44a0468054d6c3de47; no voters: a29cd590999b4f32849ff5689cd53c11
I20240501 08:12:28.284909 32475 consensus_queue.cc:260] T 6c9bb7ab554e4023892339cd9e0c2a75 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:28.285581 32475 ts_tablet_manager.cc:1398] T 6c9bb7ab554e4023892339cd9e0c2a75 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:28.294235 32378 raft_consensus.cc:2781] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:28.295909 32378 raft_consensus.cc:686] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Becoming Leader. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Running, Role: LEADER
I20240501 08:12:28.296293 32378 consensus_queue.cc:237] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:28.301450 27618 catalog_manager.cc:5484] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64998323a1f4482d8704bf20594d13a5 (127.21.84.135). New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:28.314851 32419 raft_consensus.cc:482] T 0ac9368b897246ac8b2f6b2f096ff3ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:28.315032 32419 raft_consensus.cc:504] T 0ac9368b897246ac8b2f6b2f096ff3ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:28.315721 32419 leader_election.cc:288] T 0ac9368b897246ac8b2f6b2f096ff3ea P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:28.316249 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac9368b897246ac8b2f6b2f096ff3ea" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:12:28.316478 28021 raft_consensus.cc:2454] T 0ac9368b897246ac8b2f6b2f096ff3ea P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:12:28.316722 30670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac9368b897246ac8b2f6b2f096ff3ea" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:28.316875 27696 leader_election.cc:302] T 0ac9368b897246ac8b2f6b2f096ff3ea P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:12:28.316998 30670 raft_consensus.cc:2454] T 0ac9368b897246ac8b2f6b2f096ff3ea P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:12:28.317158 32419 raft_consensus.cc:2781] T 0ac9368b897246ac8b2f6b2f096ff3ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:28.317303 32419 raft_consensus.cc:482] T 0ac9368b897246ac8b2f6b2f096ff3ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:28.317437 32419 raft_consensus.cc:3037] T 0ac9368b897246ac8b2f6b2f096ff3ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:28.319718 32419 raft_consensus.cc:504] T 0ac9368b897246ac8b2f6b2f096ff3ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:28.320263 32419 leader_election.cc:288] T 0ac9368b897246ac8b2f6b2f096ff3ea P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Requested vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:28.320767 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac9368b897246ac8b2f6b2f096ff3ea" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:12:28.320986 28021 raft_consensus.cc:3037] T 0ac9368b897246ac8b2f6b2f096ff3ea P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:28.321004 30670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac9368b897246ac8b2f6b2f096ff3ea" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:28.321213 30670 raft_consensus.cc:3037] T 0ac9368b897246ac8b2f6b2f096ff3ea P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:28.323525 28021 raft_consensus.cc:2454] T 0ac9368b897246ac8b2f6b2f096ff3ea P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:12:28.323529 30670 raft_consensus.cc:2454] T 0ac9368b897246ac8b2f6b2f096ff3ea P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:12:28.323977 27696 leader_election.cc:302] T 0ac9368b897246ac8b2f6b2f096ff3ea P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:12:28.324546 32419 raft_consensus.cc:2781] T 0ac9368b897246ac8b2f6b2f096ff3ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:28.324859 32419 raft_consensus.cc:686] T 0ac9368b897246ac8b2f6b2f096ff3ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:12:28.325146 32419 consensus_queue.cc:237] T 0ac9368b897246ac8b2f6b2f096ff3ea P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:28.329077 27618 catalog_manager.cc:5484] T 0ac9368b897246ac8b2f6b2f096ff3ea P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:28.366731 27618 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:28.373293 30803 raft_consensus.cc:1261] T 94210ebbc379459d8b6e24237d33bb73 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:28.373530 30670 raft_consensus.cc:1261] T 94210ebbc379459d8b6e24237d33bb73 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:28.373950 32451 consensus_queue.cc:1035] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:28.374193 32458 consensus_queue.cc:1035] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:28.378720 32491 tablet.cc:1722] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:28.379513 32491 tablet.cc:1620] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6: MemRowSet was empty: no flush needed.
I20240501 08:12:28.380662 27616 catalog_manager.cc:5576] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933) does not have the latest schema for tablet 94210ebbc379459d8b6e24237d33bb73 (table rebalancer_dynamic_table [id=ef4f42d7d291452da72bd7f47df1e35c]). Expected version 1 got 0
I20240501 08:12:28.382094 27618 catalog_manager.cc:5576] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503) does not have the latest schema for tablet 94210ebbc379459d8b6e24237d33bb73 (table rebalancer_dynamic_table [id=ef4f42d7d291452da72bd7f47df1e35c]). Expected version 1 got 0
I20240501 08:12:28.385787 32492 tablet.cc:1722] T 94210ebbc379459d8b6e24237d33bb73 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:28.385804 32493 tablet.cc:1722] T 94210ebbc379459d8b6e24237d33bb73 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:28.386480 32493 tablet.cc:1620] T 94210ebbc379459d8b6e24237d33bb73 P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:28.386754 27620 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=ef4f42d7d291452da72bd7f47df1e35c] alter complete (version 1)
I20240501 08:12:28.387251 32492 tablet.cc:1620] T 94210ebbc379459d8b6e24237d33bb73 P 64998323a1f4482d8704bf20594d13a5: MemRowSet was empty: no flush needed.
I20240501 08:12:28.391680 27620 catalog_manager.cc:5576] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933) does not have the latest schema for tablet 94210ebbc379459d8b6e24237d33bb73 (table rebalancer_dynamic_table [id=ef4f42d7d291452da72bd7f47df1e35c]). Expected version 1 got 0
I20240501 08:12:28.399976 27616 catalog_manager.cc:5576] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503) does not have the latest schema for tablet 94210ebbc379459d8b6e24237d33bb73 (table rebalancer_dynamic_table [id=ef4f42d7d291452da72bd7f47df1e35c]). Expected version 1 got 0
I20240501 08:12:28.500021 27616 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:28.508198 32491 tablet.cc:1722] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:28.508709 32491 tablet.cc:1620] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6: MemRowSet was empty: no flush needed.
I20240501 08:12:28.511938 32309 raft_consensus.cc:482] T 6c9bb7ab554e4023892339cd9e0c2a75 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:28.512118 32309 raft_consensus.cc:504] T 6c9bb7ab554e4023892339cd9e0c2a75 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:28.512853 32309 leader_election.cc:288] T 6c9bb7ab554e4023892339cd9e0c2a75 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:28.513492 30670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c9bb7ab554e4023892339cd9e0c2a75" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:28.513808 30670 raft_consensus.cc:2379] T 6c9bb7ab554e4023892339cd9e0c2a75 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a29cd590999b4f32849ff5689cd53c11 in current term 1: Already voted for candidate 64998323a1f4482d8704bf20594d13a5 in this term.
I20240501 08:12:28.514053 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c9bb7ab554e4023892339cd9e0c2a75" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:28.514160 32492 tablet.cc:1722] T 94210ebbc379459d8b6e24237d33bb73 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:28.514658 32492 tablet.cc:1620] T 94210ebbc379459d8b6e24237d33bb73 P 64998323a1f4482d8704bf20594d13a5: MemRowSet was empty: no flush needed.
I20240501 08:12:28.514806 32493 tablet.cc:1722] T 94210ebbc379459d8b6e24237d33bb73 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:28.515297 32493 tablet.cc:1620] T 94210ebbc379459d8b6e24237d33bb73 P 9a022e1a501f4d44a0468054d6c3de47: MemRowSet was empty: no flush needed.
I20240501 08:12:28.516722 30871 leader_election.cc:302] T 6c9bb7ab554e4023892339cd9e0c2a75 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a29cd590999b4f32849ff5689cd53c11; no voters: 64998323a1f4482d8704bf20594d13a5, 9a022e1a501f4d44a0468054d6c3de47
I20240501 08:12:28.517522 32309 raft_consensus.cc:3037] T 6c9bb7ab554e4023892339cd9e0c2a75 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:28.520658 32309 raft_consensus.cc:2726] T 6c9bb7ab554e4023892339cd9e0c2a75 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:28.525465 27620 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=ef4f42d7d291452da72bd7f47df1e35c] alter complete (version 2)
I20240501 08:12:28.631067 27616 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:28.638425 32491 tablet.cc:1722] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:28.639115 32492 tablet.cc:1722] T 94210ebbc379459d8b6e24237d33bb73 P 64998323a1f4482d8704bf20594d13a5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:28.639894 32493 tablet.cc:1722] T 94210ebbc379459d8b6e24237d33bb73 P 9a022e1a501f4d44a0468054d6c3de47: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:28.649329 27616 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table_new_name [id=ef4f42d7d291452da72bd7f47df1e35c] alter complete (version 3)
I20240501 08:12:28.716315 32378 consensus_queue.cc:1035] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:28.722402 32302 consensus_queue.cc:1035] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:28.745203 32444 consensus_queue.cc:1035] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:28.752221 32419 consensus_queue.cc:1035] T 0ac9368b897246ac8b2f6b2f096ff3ea P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:28.753116 27616 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:28.753358 27616 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:28.756979 27616 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 94210ebbc379459d8b6e24237d33bb73
I20240501 08:12:28.758669 32444 consensus_queue.cc:1035] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:28.761129 30781 tablet_service.cc:1514] Processing DeleteTablet for tablet 94210ebbc379459d8b6e24237d33bb73 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:28 UTC) from {username='slave'} at 127.0.0.1:38952
I20240501 08:12:28.761741 28132 tablet_service.cc:1514] Processing DeleteTablet for tablet 94210ebbc379459d8b6e24237d33bb73 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:28 UTC) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:28.772066 32346 consensus_queue.cc:1035] T 0ac9368b897246ac8b2f6b2f096ff3ea P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:28.776738 30650 tablet_service.cc:1514] Processing DeleteTablet for tablet 94210ebbc379459d8b6e24237d33bb73 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:28 UTC) from {username='slave'} at 127.0.0.1:57156
I20240501 08:12:28.794544 32508 tablet_replica.cc:330] T 94210ebbc379459d8b6e24237d33bb73 P 9a022e1a501f4d44a0468054d6c3de47: stopping tablet replica
I20240501 08:12:28.794967 32508 raft_consensus.cc:2229] T 94210ebbc379459d8b6e24237d33bb73 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:28.795451 32508 raft_consensus.cc:2258] T 94210ebbc379459d8b6e24237d33bb73 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:28.796722 32508 ts_tablet_manager.cc:1875] T 94210ebbc379459d8b6e24237d33bb73 P 9a022e1a501f4d44a0468054d6c3de47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:28.797859 32504 tablet_replica.cc:330] T 94210ebbc379459d8b6e24237d33bb73 P 64998323a1f4482d8704bf20594d13a5: stopping tablet replica
I20240501 08:12:28.798251 32504 raft_consensus.cc:2229] T 94210ebbc379459d8b6e24237d33bb73 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:28.798564 32504 raft_consensus.cc:2258] T 94210ebbc379459d8b6e24237d33bb73 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:28.799391 32505 tablet_replica.cc:330] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:28.799758 32505 raft_consensus.cc:2229] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:28.800393 32505 raft_consensus.cc:2258] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:28.801712 32504 ts_tablet_manager.cc:1875] T 94210ebbc379459d8b6e24237d33bb73 P 64998323a1f4482d8704bf20594d13a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:28.805517 32508 ts_tablet_manager.cc:1888] T 94210ebbc379459d8b6e24237d33bb73 P 9a022e1a501f4d44a0468054d6c3de47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:28.806270 32505 ts_tablet_manager.cc:1875] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:28.806519 32508 log.cc:1196] T 94210ebbc379459d8b6e24237d33bb73 P 9a022e1a501f4d44a0468054d6c3de47: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/wals/94210ebbc379459d8b6e24237d33bb73
I20240501 08:12:28.807119 32508 ts_tablet_manager.cc:1909] T 94210ebbc379459d8b6e24237d33bb73 P 9a022e1a501f4d44a0468054d6c3de47: Deleting consensus metadata
I20240501 08:12:28.815032 27604 catalog_manager.cc:4830] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): tablet 94210ebbc379459d8b6e24237d33bb73 (table rebalancer_dynamic_table_new_name [id=ef4f42d7d291452da72bd7f47df1e35c]) successfully deleted
I20240501 08:12:28.815725 32505 ts_tablet_manager.cc:1888] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:28.815945 32505 log.cc:1196] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/94210ebbc379459d8b6e24237d33bb73
I20240501 08:12:28.816434 32505 ts_tablet_manager.cc:1909] T 94210ebbc379459d8b6e24237d33bb73 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting consensus metadata
I20240501 08:12:28.817672 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet 94210ebbc379459d8b6e24237d33bb73 (table rebalancer_dynamic_table_new_name [id=ef4f42d7d291452da72bd7f47df1e35c]) successfully deleted
I20240501 08:12:28.823598 32504 ts_tablet_manager.cc:1888] T 94210ebbc379459d8b6e24237d33bb73 P 64998323a1f4482d8704bf20594d13a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:28.823738 32504 log.cc:1196] T 94210ebbc379459d8b6e24237d33bb73 P 64998323a1f4482d8704bf20594d13a5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal/wals/94210ebbc379459d8b6e24237d33bb73
I20240501 08:12:28.824159 32504 ts_tablet_manager.cc:1909] T 94210ebbc379459d8b6e24237d33bb73 P 64998323a1f4482d8704bf20594d13a5: Deleting consensus metadata
I20240501 08:12:28.825037 27605 catalog_manager.cc:4830] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): tablet 94210ebbc379459d8b6e24237d33bb73 (table rebalancer_dynamic_table_new_name [id=ef4f42d7d291452da72bd7f47df1e35c]) successfully deleted
I20240501 08:12:28.845883 27620 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_extra_table_52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:28.854594 28132 tablet_service.cc:1467] Processing CreateTablet for tablet 9e22cda990da493d9aa7d9401552cb6e (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=238033141b1b4b5b847f037773a4022f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:28.855111 28132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e22cda990da493d9aa7d9401552cb6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:28.855093 27871 tablet_service.cc:1467] Processing CreateTablet for tablet 9e22cda990da493d9aa7d9401552cb6e (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=238033141b1b4b5b847f037773a4022f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:28.855347 28002 tablet_service.cc:1467] Processing CreateTablet for tablet 2e1dd7a48a574ffcbea936af5b0fe192 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=238033141b1b4b5b847f037773a4022f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:28.855549 27871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e22cda990da493d9aa7d9401552cb6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:28.855731 28002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e1dd7a48a574ffcbea936af5b0fe192. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:28.861122 28131 tablet_service.cc:1467] Processing CreateTablet for tablet 2e1dd7a48a574ffcbea936af5b0fe192 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=238033141b1b4b5b847f037773a4022f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:28.861603 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e1dd7a48a574ffcbea936af5b0fe192. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:28.861661 30784 tablet_service.cc:1467] Processing CreateTablet for tablet 2e1dd7a48a574ffcbea936af5b0fe192 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=238033141b1b4b5b847f037773a4022f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:28.862357 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e1dd7a48a574ffcbea936af5b0fe192. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:28.864346 30781 tablet_service.cc:1467] Processing CreateTablet for tablet 9e22cda990da493d9aa7d9401552cb6e (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=238033141b1b4b5b847f037773a4022f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:28.864715 30781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e22cda990da493d9aa7d9401552cb6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:28.864902 27620 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:28.874971 30517 tablet_service.cc:1467] Processing CreateTablet for tablet 8c3d0b4ae0404bdfada8a86b23db9d30 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=238033141b1b4b5b847f037773a4022f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:28.875449 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c3d0b4ae0404bdfada8a86b23db9d30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:28.877151 27741 tablet_service.cc:1467] Processing CreateTablet for tablet 8c3d0b4ae0404bdfada8a86b23db9d30 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=238033141b1b4b5b847f037773a4022f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:28.877691 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c3d0b4ae0404bdfada8a86b23db9d30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:28.878432 30915 tablet_service.cc:1467] Processing CreateTablet for tablet 8c3d0b4ae0404bdfada8a86b23db9d30 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=238033141b1b4b5b847f037773a4022f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:28.878965 30915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c3d0b4ae0404bdfada8a86b23db9d30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:28.880789 32513 tablet_bootstrap.cc:492] T 9e22cda990da493d9aa7d9401552cb6e P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:28.886241 32513 tablet_bootstrap.cc:654] T 9e22cda990da493d9aa7d9401552cb6e P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:28.891228 32517 tablet_bootstrap.cc:492] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:28.897099 32517 tablet_bootstrap.cc:654] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:28.897804 32512 tablet_bootstrap.cc:492] T 9e22cda990da493d9aa7d9401552cb6e P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:28.900041 28000 tablet_service.cc:1467] Processing CreateTablet for tablet 15d968429db8425b90ab086725f95f24 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=688c303c38a1497aa1524193091b5e7a]), partition=
I20240501 08:12:28.900472 28000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15d968429db8425b90ab086725f95f24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:28.900717 30514 tablet_service.cc:1467] Processing CreateTablet for tablet 15d968429db8425b90ab086725f95f24 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=688c303c38a1497aa1524193091b5e7a]), partition=
I20240501 08:12:28.901131 30514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15d968429db8425b90ab086725f95f24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:28.909149 27871 tablet_service.cc:1467] Processing CreateTablet for tablet 15d968429db8425b90ab086725f95f24 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=688c303c38a1497aa1524193091b5e7a]), partition=
I20240501 08:12:28.910841 32512 tablet_bootstrap.cc:654] T 9e22cda990da493d9aa7d9401552cb6e P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:28.920200 32518 tablet_bootstrap.cc:492] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:28.922681 32518 tablet_bootstrap.cc:654] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:28.929678 32512 tablet_bootstrap.cc:492] T 9e22cda990da493d9aa7d9401552cb6e P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:12:28.929957 32512 ts_tablet_manager.cc:1367] T 9e22cda990da493d9aa7d9401552cb6e P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.000s
I20240501 08:12:28.930835 32512 raft_consensus.cc:348] T 9e22cda990da493d9aa7d9401552cb6e P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:28.931169 32512 raft_consensus.cc:374] T 9e22cda990da493d9aa7d9401552cb6e P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:28.931311 32512 raft_consensus.cc:729] T 9e22cda990da493d9aa7d9401552cb6e P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:12:28.931670 32512 consensus_queue.cc:260] T 9e22cda990da493d9aa7d9401552cb6e P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:28.932461 32512 ts_tablet_manager.cc:1398] T 9e22cda990da493d9aa7d9401552cb6e P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:28.932862 32512 tablet_bootstrap.cc:492] T 2e1dd7a48a574ffcbea936af5b0fe192 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:28.934520 32512 tablet_bootstrap.cc:654] T 2e1dd7a48a574ffcbea936af5b0fe192 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:28.937541 27871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15d968429db8425b90ab086725f95f24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:28.950871 32518 tablet_bootstrap.cc:492] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:12:28.951118 32518 ts_tablet_manager.cc:1367] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.004s
I20240501 08:12:28.951894 32518 raft_consensus.cc:348] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:28.952207 32518 raft_consensus.cc:374] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:28.952359 32518 raft_consensus.cc:729] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:12:28.952729 32518 consensus_queue.cc:260] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:28.952880 32512 tablet_bootstrap.cc:492] T 2e1dd7a48a574ffcbea936af5b0fe192 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:12:28.953083 32512 ts_tablet_manager.cc:1367] T 2e1dd7a48a574ffcbea936af5b0fe192 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20240501 08:12:28.953577 32518 ts_tablet_manager.cc:1398] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:28.953979 32512 raft_consensus.cc:348] T 2e1dd7a48a574ffcbea936af5b0fe192 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:28.954245 32512 raft_consensus.cc:374] T 2e1dd7a48a574ffcbea936af5b0fe192 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:28.954387 32512 raft_consensus.cc:729] T 2e1dd7a48a574ffcbea936af5b0fe192 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:12:28.967319 32512 consensus_queue.cc:260] T 2e1dd7a48a574ffcbea936af5b0fe192 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:28.968170 32512 ts_tablet_manager.cc:1398] T 2e1dd7a48a574ffcbea936af5b0fe192 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20240501 08:12:28.973381 32520 tablet_bootstrap.cc:492] T 8c3d0b4ae0404bdfada8a86b23db9d30 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:28.980244 32520 tablet_bootstrap.cc:654] T 8c3d0b4ae0404bdfada8a86b23db9d30 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:28.986529 32515 tablet_bootstrap.cc:492] T 2e1dd7a48a574ffcbea936af5b0fe192 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:28.988821 32515 tablet_bootstrap.cc:654] T 2e1dd7a48a574ffcbea936af5b0fe192 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:28.989485 32514 tablet_bootstrap.cc:492] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:28.991786 32514 tablet_bootstrap.cc:654] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:28.993741 32517 tablet_bootstrap.cc:492] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:28.993901 32517 ts_tablet_manager.cc:1367] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.103s user 0.010s sys 0.020s
I20240501 08:12:28.996182 32513 tablet_bootstrap.cc:492] T 9e22cda990da493d9aa7d9401552cb6e P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:12:28.996415 32513 ts_tablet_manager.cc:1367] T 9e22cda990da493d9aa7d9401552cb6e P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.116s user 0.026s sys 0.010s
I20240501 08:12:28.996357 32517 raft_consensus.cc:348] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:28.996621 32517 raft_consensus.cc:374] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:28.996747 32517 raft_consensus.cc:729] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:28.999109 32513 raft_consensus.cc:348] T 9e22cda990da493d9aa7d9401552cb6e P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:28.999411 32513 raft_consensus.cc:374] T 9e22cda990da493d9aa7d9401552cb6e P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:28.999563 32513 raft_consensus.cc:729] T 9e22cda990da493d9aa7d9401552cb6e P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:12:28.997314 32517 consensus_queue.cc:260] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:28.999922 32513 consensus_queue.cc:260] T 9e22cda990da493d9aa7d9401552cb6e P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:29.001859 32513 ts_tablet_manager.cc:1398] T 9e22cda990da493d9aa7d9401552cb6e P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 08:12:29.002216 32513 tablet_bootstrap.cc:492] T 15d968429db8425b90ab086725f95f24 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:29.004160 32513 tablet_bootstrap.cc:654] T 15d968429db8425b90ab086725f95f24 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:29.004982 32520 tablet_bootstrap.cc:492] T 8c3d0b4ae0404bdfada8a86b23db9d30 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:29.005215 32520 ts_tablet_manager.cc:1367] T 8c3d0b4ae0404bdfada8a86b23db9d30 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20240501 08:12:29.006134 32520 raft_consensus.cc:348] T 8c3d0b4ae0404bdfada8a86b23db9d30 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:29.006448 32520 raft_consensus.cc:374] T 8c3d0b4ae0404bdfada8a86b23db9d30 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:29.006620 32520 raft_consensus.cc:729] T 8c3d0b4ae0404bdfada8a86b23db9d30 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:29.007499 32520 consensus_queue.cc:260] T 8c3d0b4ae0404bdfada8a86b23db9d30 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:29.008312 32520 ts_tablet_manager.cc:1398] T 8c3d0b4ae0404bdfada8a86b23db9d30 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:29.010627 32514 tablet_bootstrap.cc:492] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:29.010828 32514 ts_tablet_manager.cc:1367] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.005s
I20240501 08:12:29.011641 32514 raft_consensus.cc:348] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:29.011927 32514 raft_consensus.cc:374] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:29.012066 32514 raft_consensus.cc:729] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:29.012338 32514 consensus_queue.cc:260] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:29.012754 32515 tablet_bootstrap.cc:492] T 2e1dd7a48a574ffcbea936af5b0fe192 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:12:29.012974 32514 ts_tablet_manager.cc:1398] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:29.012972 32515 ts_tablet_manager.cc:1367] T 2e1dd7a48a574ffcbea936af5b0fe192 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.008s
I20240501 08:12:29.013999 32515 raft_consensus.cc:348] T 2e1dd7a48a574ffcbea936af5b0fe192 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:29.014451 32515 raft_consensus.cc:374] T 2e1dd7a48a574ffcbea936af5b0fe192 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:29.014604 32515 raft_consensus.cc:729] T 2e1dd7a48a574ffcbea936af5b0fe192 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:12:29.014923 32515 consensus_queue.cc:260] T 2e1dd7a48a574ffcbea936af5b0fe192 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:29.016917 32515 ts_tablet_manager.cc:1398] T 2e1dd7a48a574ffcbea936af5b0fe192 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:29.017226 32515 tablet_bootstrap.cc:492] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:29.019017 32515 tablet_bootstrap.cc:654] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:29.024063 32517 ts_tablet_manager.cc:1398] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.030s user 0.002s sys 0.000s
I20240501 08:12:29.025928 32514 tablet_bootstrap.cc:492] T 9e22cda990da493d9aa7d9401552cb6e P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:29.033825 32515 tablet_bootstrap.cc:492] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:12:29.034174 32515 ts_tablet_manager.cc:1367] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.005s
I20240501 08:12:29.034438 32514 tablet_bootstrap.cc:654] T 9e22cda990da493d9aa7d9401552cb6e P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:29.035076 32517 tablet_bootstrap.cc:492] T 15d968429db8425b90ab086725f95f24 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:29.035306 32515 raft_consensus.cc:348] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:29.035671 32515 raft_consensus.cc:374] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:29.035810 32515 raft_consensus.cc:729] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:12:29.036218 32513 tablet_bootstrap.cc:492] T 15d968429db8425b90ab086725f95f24 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:12:29.036134 32515 consensus_queue.cc:260] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:29.036417 32513 ts_tablet_manager.cc:1367] T 15d968429db8425b90ab086725f95f24 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.034s user 0.010s sys 0.012s
I20240501 08:12:29.037266 32513 raft_consensus.cc:348] T 15d968429db8425b90ab086725f95f24 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:29.037611 32513 raft_consensus.cc:374] T 15d968429db8425b90ab086725f95f24 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:29.037722 32513 raft_consensus.cc:729] T 15d968429db8425b90ab086725f95f24 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:12:29.038023 32513 consensus_queue.cc:260] T 15d968429db8425b90ab086725f95f24 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:29.038880 32515 ts_tablet_manager.cc:1398] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 08:12:29.039911 32513 ts_tablet_manager.cc:1398] T 15d968429db8425b90ab086725f95f24 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 08:12:29.042042 32514 tablet_bootstrap.cc:492] T 9e22cda990da493d9aa7d9401552cb6e P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:29.042291 32514 ts_tablet_manager.cc:1367] T 9e22cda990da493d9aa7d9401552cb6e P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20240501 08:12:29.043500 32514 raft_consensus.cc:348] T 9e22cda990da493d9aa7d9401552cb6e P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:29.044366 32514 raft_consensus.cc:374] T 9e22cda990da493d9aa7d9401552cb6e P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:29.044504 32514 raft_consensus.cc:729] T 9e22cda990da493d9aa7d9401552cb6e P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:29.044780 32514 consensus_queue.cc:260] T 9e22cda990da493d9aa7d9401552cb6e P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:29.045432 32514 ts_tablet_manager.cc:1398] T 9e22cda990da493d9aa7d9401552cb6e P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:29.046612 32517 tablet_bootstrap.cc:654] T 15d968429db8425b90ab086725f95f24 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:29.062985 32517 tablet_bootstrap.cc:492] T 15d968429db8425b90ab086725f95f24 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:29.063174 32517 ts_tablet_manager.cc:1367] T 15d968429db8425b90ab086725f95f24 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.005s
I20240501 08:12:29.063912 32517 raft_consensus.cc:348] T 15d968429db8425b90ab086725f95f24 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:29.064160 32517 raft_consensus.cc:374] T 15d968429db8425b90ab086725f95f24 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:29.064491 32517 raft_consensus.cc:729] T 15d968429db8425b90ab086725f95f24 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:29.064740 32517 consensus_queue.cc:260] T 15d968429db8425b90ab086725f95f24 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:29.065338 32517 ts_tablet_manager.cc:1398] T 15d968429db8425b90ab086725f95f24 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:29.083678 32378 raft_consensus.cc:482] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:29.083887 32378 raft_consensus.cc:504] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:29.084545 32378 leader_election.cc:288] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:29.085136 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e1dd7a48a574ffcbea936af5b0fe192" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:12:29.085351 28150 raft_consensus.cc:2454] T 2e1dd7a48a574ffcbea936af5b0fe192 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:29.085390 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e1dd7a48a574ffcbea936af5b0fe192" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:12:29.085592 28022 raft_consensus.cc:2454] T 2e1dd7a48a574ffcbea936af5b0fe192 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:29.085790 30737 leader_election.cc:302] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:12:29.086117 32378 raft_consensus.cc:2781] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:29.086230 32378 raft_consensus.cc:482] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:29.086333 32378 raft_consensus.cc:3037] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:29.088054 32378 raft_consensus.cc:504] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:29.088668 32378 leader_election.cc:288] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Requested vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:29.089087 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e1dd7a48a574ffcbea936af5b0fe192" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:12:29.089075 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e1dd7a48a574ffcbea936af5b0fe192" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:12:29.089280 28022 raft_consensus.cc:3037] T 2e1dd7a48a574ffcbea936af5b0fe192 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:29.089314 28150 raft_consensus.cc:3037] T 2e1dd7a48a574ffcbea936af5b0fe192 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:29.091599 28022 raft_consensus.cc:2454] T 2e1dd7a48a574ffcbea936af5b0fe192 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:29.092064 30739 leader_election.cc:302] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:12:29.092353 32378 raft_consensus.cc:2781] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:29.092595 28150 raft_consensus.cc:2454] T 2e1dd7a48a574ffcbea936af5b0fe192 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:29.094075 32444 raft_consensus.cc:482] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:29.094264 32444 raft_consensus.cc:504] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:29.094863 32378 raft_consensus.cc:686] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Becoming Leader. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Running, Role: LEADER
I20240501 08:12:29.094980 32444 leader_election.cc:288] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:29.095142 32378 consensus_queue.cc:237] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:29.096036 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3d0b4ae0404bdfada8a86b23db9d30" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:29.096251 30936 raft_consensus.cc:2454] T 8c3d0b4ae0404bdfada8a86b23db9d30 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 0.
I20240501 08:12:29.097358 30472 leader_election.cc:302] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:29.098512 32444 raft_consensus.cc:2781] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:29.098613 32444 raft_consensus.cc:482] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:29.098732 32444 raft_consensus.cc:3037] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:29.099759 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3d0b4ae0404bdfada8a86b23db9d30" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:12:29.099990 27759 raft_consensus.cc:2454] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 0.
I20240501 08:12:29.101469 32444 raft_consensus.cc:504] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:29.101982 27618 catalog_manager.cc:5484] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64998323a1f4482d8704bf20594d13a5 (127.21.84.135). New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:29.102649 32444 leader_election.cc:288] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 election: Requested vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:29.102887 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3d0b4ae0404bdfada8a86b23db9d30" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:12:29.103111 27759 raft_consensus.cc:3037] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:29.103942 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3d0b4ae0404bdfada8a86b23db9d30" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:29.104151 30936 raft_consensus.cc:3037] T 8c3d0b4ae0404bdfada8a86b23db9d30 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:29.105999 27759 raft_consensus.cc:2454] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 1.
I20240501 08:12:29.106609 30472 leader_election.cc:302] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 6331f77dae9d452184e8cd0cbf5f1b43; no voters:
I20240501 08:12:29.106838 30936 raft_consensus.cc:2454] T 8c3d0b4ae0404bdfada8a86b23db9d30 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 1.
I20240501 08:12:29.106961 32444 raft_consensus.cc:2781] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:29.107124 32444 raft_consensus.cc:686] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Becoming Leader. State: Replica: 31811708c6774037b82df44024b229b6, State: Running, Role: LEADER
I20240501 08:12:29.107620 32444 consensus_queue.cc:237] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:29.124078 27618 catalog_manager.cc:5484] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 31811708c6774037b82df44024b229b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31811708c6774037b82df44024b229b6 (127.21.84.133). New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:29.145957 32278 raft_consensus.cc:482] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:29.146144 32278 raft_consensus.cc:504] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:29.146838 32278 leader_election.cc:288] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:29.147315 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15d968429db8425b90ab086725f95f24" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:12:29.147423 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15d968429db8425b90ab086725f95f24" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:29.147563 27889 raft_consensus.cc:2454] T 15d968429db8425b90ab086725f95f24 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 0.
I20240501 08:12:29.147665 30536 raft_consensus.cc:2454] T 15d968429db8425b90ab086725f95f24 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 0.
I20240501 08:12:29.148002 27958 leader_election.cc:302] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aad81047009a4ffcbc450b60030583b5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:29.148263 32278 raft_consensus.cc:2781] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:29.148377 32278 raft_consensus.cc:482] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:29.148461 32278 raft_consensus.cc:3037] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:29.150854 32278 raft_consensus.cc:504] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:29.151472 32278 leader_election.cc:288] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Requested vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:29.151799 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15d968429db8425b90ab086725f95f24" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:12:29.152016 27889 raft_consensus.cc:3037] T 15d968429db8425b90ab086725f95f24 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:29.151980 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15d968429db8425b90ab086725f95f24" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:29.152150 30536 raft_consensus.cc:3037] T 15d968429db8425b90ab086725f95f24 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:29.154261 27889 raft_consensus.cc:2454] T 15d968429db8425b90ab086725f95f24 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 1.
I20240501 08:12:29.154693 27958 leader_election.cc:302] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aad81047009a4ffcbc450b60030583b5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:29.154760 30536 raft_consensus.cc:2454] T 15d968429db8425b90ab086725f95f24 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 1.
I20240501 08:12:29.155936 32534 raft_consensus.cc:2781] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:29.156075 32534 raft_consensus.cc:686] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Becoming Leader. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Running, Role: LEADER
I20240501 08:12:29.156451 32534 consensus_queue.cc:237] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:29.168325 27618 catalog_manager.cc:5484] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to aad81047009a4ffcbc450b60030583b5 (127.21.84.131). New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:29.203372 32451 raft_consensus.cc:482] T 9e22cda990da493d9aa7d9401552cb6e P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:29.203589 32451 raft_consensus.cc:504] T 9e22cda990da493d9aa7d9401552cb6e P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:29.204339 32451 leader_election.cc:288] T 9e22cda990da493d9aa7d9401552cb6e P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:12:29.204782 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e22cda990da493d9aa7d9401552cb6e" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:29.204825 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e22cda990da493d9aa7d9401552cb6e" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:12:29.205005 30804 raft_consensus.cc:2454] T 9e22cda990da493d9aa7d9401552cb6e P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:12:29.205044 27889 raft_consensus.cc:2454] T 9e22cda990da493d9aa7d9401552cb6e P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:12:29.205473 28088 leader_election.cc:302] T 9e22cda990da493d9aa7d9401552cb6e P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:29.205765 32293 raft_consensus.cc:2781] T 9e22cda990da493d9aa7d9401552cb6e P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:29.205865 32293 raft_consensus.cc:482] T 9e22cda990da493d9aa7d9401552cb6e P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:29.205994 32293 raft_consensus.cc:3037] T 9e22cda990da493d9aa7d9401552cb6e P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:29.208492 32293 raft_consensus.cc:504] T 9e22cda990da493d9aa7d9401552cb6e P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:29.209086 32293 leader_election.cc:288] T 9e22cda990da493d9aa7d9401552cb6e P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:12:29.209467 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e22cda990da493d9aa7d9401552cb6e" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:29.209688 30804 raft_consensus.cc:3037] T 9e22cda990da493d9aa7d9401552cb6e P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:29.209656 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e22cda990da493d9aa7d9401552cb6e" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:12:29.209846 27889 raft_consensus.cc:3037] T 9e22cda990da493d9aa7d9401552cb6e P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:29.212062 30804 raft_consensus.cc:2454] T 9e22cda990da493d9aa7d9401552cb6e P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:12:29.212095 27889 raft_consensus.cc:2454] T 9e22cda990da493d9aa7d9401552cb6e P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:12:29.212620 28086 leader_election.cc:302] T 9e22cda990da493d9aa7d9401552cb6e P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:12:29.212880 32293 raft_consensus.cc:2781] T 9e22cda990da493d9aa7d9401552cb6e P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:29.213074 32293 raft_consensus.cc:686] T 9e22cda990da493d9aa7d9401552cb6e P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:12:29.213376 32293 consensus_queue.cc:237] T 9e22cda990da493d9aa7d9401552cb6e P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:29.221451 27618 catalog_manager.cc:5484] T 9e22cda990da493d9aa7d9401552cb6e P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:29.308688 27618 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:29.314417 30536 raft_consensus.cc:1261] T 15d968429db8425b90ab086725f95f24 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:29.314476 27889 raft_consensus.cc:1261] T 15d968429db8425b90ab086725f95f24 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:29.315104 32533 consensus_queue.cc:1035] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:29.315600 32533 consensus_queue.cc:1035] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:29.321098 27618 catalog_manager.cc:5576] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293) does not have the latest schema for tablet 15d968429db8425b90ab086725f95f24 (table rebalancer_dynamic_table [id=688c303c38a1497aa1524193091b5e7a]). Expected version 1 got 0
I20240501 08:12:29.333251 32543 tablet.cc:1722] T 15d968429db8425b90ab086725f95f24 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:29.334116 32542 tablet.cc:1722] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:29.335472 32543 tablet.cc:1620] T 15d968429db8425b90ab086725f95f24 P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:29.335513 32542 tablet.cc:1620] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5: MemRowSet was empty: no flush needed.
I20240501 08:12:29.338395 27618 catalog_manager.cc:5576] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293) does not have the latest schema for tablet 15d968429db8425b90ab086725f95f24 (table rebalancer_dynamic_table [id=688c303c38a1497aa1524193091b5e7a]). Expected version 1 got 0
I20240501 08:12:29.341912 27618 catalog_manager.cc:5576] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503) does not have the latest schema for tablet 15d968429db8425b90ab086725f95f24 (table rebalancer_dynamic_table [id=688c303c38a1497aa1524193091b5e7a]). Expected version 1 got 0
I20240501 08:12:29.343230 27620 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=688c303c38a1497aa1524193091b5e7a] alter complete (version 1)
I20240501 08:12:29.343987 32544 tablet.cc:1722] T 15d968429db8425b90ab086725f95f24 P f1525cbc82a24d13b65f136695c903af: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:29.344677 32544 tablet.cc:1620] T 15d968429db8425b90ab086725f95f24 P f1525cbc82a24d13b65f136695c903af: MemRowSet was empty: no flush needed.
I20240501 08:12:29.445560 27618 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:29.453811 32542 tablet.cc:1722] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:29.454952 32542 tablet.cc:1620] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5: MemRowSet was empty: no flush needed.
I20240501 08:12:29.455929 32544 tablet.cc:1722] T 15d968429db8425b90ab086725f95f24 P f1525cbc82a24d13b65f136695c903af: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:29.456415 32544 tablet.cc:1620] T 15d968429db8425b90ab086725f95f24 P f1525cbc82a24d13b65f136695c903af: MemRowSet was empty: no flush needed.
I20240501 08:12:29.456918 32543 tablet.cc:1722] T 15d968429db8425b90ab086725f95f24 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:29.457429 32543 tablet.cc:1620] T 15d968429db8425b90ab086725f95f24 P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:29.469508 27620 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=688c303c38a1497aa1524193091b5e7a] alter complete (version 2)
I20240501 08:12:29.512101 32500 consensus_queue.cc:1035] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:29.516353 32545 consensus_queue.cc:1035] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:29.574337 32378 consensus_queue.cc:1035] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:29.576164 27618 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:29.590296 32542 tablet.cc:1722] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:29.593477 32543 tablet.cc:1722] T 15d968429db8425b90ab086725f95f24 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:29.595217 32544 tablet.cc:1722] T 15d968429db8425b90ab086725f95f24 P f1525cbc82a24d13b65f136695c903af: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:29.603996 32467 consensus_queue.cc:1035] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:29.608610 27618 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table_new_name [id=688c303c38a1497aa1524193091b5e7a] alter complete (version 3)
I20240501 08:12:29.713364 27618 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:29.713588 27618 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:29.716518 27618 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 15d968429db8425b90ab086725f95f24
I20240501 08:12:29.717360 28002 tablet_service.cc:1514] Processing DeleteTablet for tablet 15d968429db8425b90ab086725f95f24 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:29 UTC) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:29.717510 30517 tablet_service.cc:1514] Processing DeleteTablet for tablet 15d968429db8425b90ab086725f95f24 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:29 UTC) from {username='slave'} at 127.0.0.1:45078
I20240501 08:12:29.717608 27871 tablet_service.cc:1514] Processing DeleteTablet for tablet 15d968429db8425b90ab086725f95f24 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:29 UTC) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:29.718304 32554 tablet_replica.cc:330] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:29.718624 32555 tablet_replica.cc:330] T 15d968429db8425b90ab086725f95f24 P 31811708c6774037b82df44024b229b6: stopping tablet replica
I20240501 08:12:29.718966 32555 raft_consensus.cc:2229] T 15d968429db8425b90ab086725f95f24 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:29.719259 32555 raft_consensus.cc:2258] T 15d968429db8425b90ab086725f95f24 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:29.720242 32554 raft_consensus.cc:2229] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:29.720595 32555 ts_tablet_manager.cc:1875] T 15d968429db8425b90ab086725f95f24 P 31811708c6774037b82df44024b229b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:29.720724 32554 raft_consensus.cc:2258] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:29.720748 32556 tablet_replica.cc:330] T 15d968429db8425b90ab086725f95f24 P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:29.720952 32556 raft_consensus.cc:2229] T 15d968429db8425b90ab086725f95f24 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:29.721200 32556 raft_consensus.cc:2258] T 15d968429db8425b90ab086725f95f24 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:29.721681 32554 ts_tablet_manager.cc:1875] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:29.722085 32556 ts_tablet_manager.cc:1875] T 15d968429db8425b90ab086725f95f24 P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:29.729354 32556 ts_tablet_manager.cc:1888] T 15d968429db8425b90ab086725f95f24 P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:29.729589 32556 log.cc:1196] T 15d968429db8425b90ab086725f95f24 P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/15d968429db8425b90ab086725f95f24
I20240501 08:12:29.729840 32555 ts_tablet_manager.cc:1888] T 15d968429db8425b90ab086725f95f24 P 31811708c6774037b82df44024b229b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:29.730123 32555 log.cc:1196] T 15d968429db8425b90ab086725f95f24 P 31811708c6774037b82df44024b229b6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/wals/15d968429db8425b90ab086725f95f24
I20240501 08:12:29.730298 32556 ts_tablet_manager.cc:1909] T 15d968429db8425b90ab086725f95f24 P f1525cbc82a24d13b65f136695c903af: Deleting consensus metadata
I20240501 08:12:29.730664 32555 ts_tablet_manager.cc:1909] T 15d968429db8425b90ab086725f95f24 P 31811708c6774037b82df44024b229b6: Deleting consensus metadata
I20240501 08:12:29.731825 27607 catalog_manager.cc:4830] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): tablet 15d968429db8425b90ab086725f95f24 (table rebalancer_dynamic_table_new_name [id=688c303c38a1497aa1524193091b5e7a]) successfully deleted
I20240501 08:12:29.732098 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet 15d968429db8425b90ab086725f95f24 (table rebalancer_dynamic_table_new_name [id=688c303c38a1497aa1524193091b5e7a]) successfully deleted
I20240501 08:12:29.732321 32554 ts_tablet_manager.cc:1888] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:29.732474 32554 log.cc:1196] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/15d968429db8425b90ab086725f95f24
I20240501 08:12:29.733026 32554 ts_tablet_manager.cc:1909] T 15d968429db8425b90ab086725f95f24 P aad81047009a4ffcbc450b60030583b5: Deleting consensus metadata
I20240501 08:12:29.734086 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet 15d968429db8425b90ab086725f95f24 (table rebalancer_dynamic_table_new_name [id=688c303c38a1497aa1524193091b5e7a]) successfully deleted
I20240501 08:12:29.792809 27618 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_extra_table_54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:29.801306 30650 tablet_service.cc:1467] Processing CreateTablet for tablet 28a1f9ac749143de951c1b41beec9875 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=20ee5c0014e146438233fd009facbf2c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:29.801831 30650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28a1f9ac749143de951c1b41beec9875. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:29.802230 30652 tablet_service.cc:1467] Processing CreateTablet for tablet 7affd631d1e444eaaf8a4c51267a204f (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=20ee5c0014e146438233fd009facbf2c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:29.802484 28132 tablet_service.cc:1467] Processing CreateTablet for tablet 6c08d494131240c1ae8180f652208325 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=20ee5c0014e146438233fd009facbf2c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:29.802611 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7affd631d1e444eaaf8a4c51267a204f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:29.802891 28132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c08d494131240c1ae8180f652208325. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:29.802793 30784 tablet_service.cc:1467] Processing CreateTablet for tablet 28a1f9ac749143de951c1b41beec9875 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=20ee5c0014e146438233fd009facbf2c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:29.803270 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28a1f9ac749143de951c1b41beec9875. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:29.806643 30517 tablet_service.cc:1467] Processing CreateTablet for tablet 6c08d494131240c1ae8180f652208325 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=20ee5c0014e146438233fd009facbf2c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:29.807124 30517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c08d494131240c1ae8180f652208325. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:29.807821 30917 tablet_service.cc:1467] Processing CreateTablet for tablet 7affd631d1e444eaaf8a4c51267a204f (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=20ee5c0014e146438233fd009facbf2c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:29.807951 30916 tablet_service.cc:1467] Processing CreateTablet for tablet 6c08d494131240c1ae8180f652208325 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=20ee5c0014e146438233fd009facbf2c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:29.808284 30917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7affd631d1e444eaaf8a4c51267a204f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:29.808579 30916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c08d494131240c1ae8180f652208325. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:29.810325 30514 tablet_service.cc:1467] Processing CreateTablet for tablet 7affd631d1e444eaaf8a4c51267a204f (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=20ee5c0014e146438233fd009facbf2c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:29.810704 30514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7affd631d1e444eaaf8a4c51267a204f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:29.814378 30915 tablet_service.cc:1467] Processing CreateTablet for tablet 28a1f9ac749143de951c1b41beec9875 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=20ee5c0014e146438233fd009facbf2c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:29.814769 30915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28a1f9ac749143de951c1b41beec9875. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:29.819368 27613 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:29.822711 32293 consensus_queue.cc:1035] T 9e22cda990da493d9aa7d9401552cb6e P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:29.861316 32562 tablet_bootstrap.cc:492] T 28a1f9ac749143de951c1b41beec9875 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:29.862814 32565 tablet_bootstrap.cc:492] T 6c08d494131240c1ae8180f652208325 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:29.863126 32557 tablet_bootstrap.cc:492] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:29.866619 32557 tablet_bootstrap.cc:654] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:29.870726 32562 tablet_bootstrap.cc:654] T 28a1f9ac749143de951c1b41beec9875 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:29.871783 27871 tablet_service.cc:1467] Processing CreateTablet for tablet 208771e4c9824fcf970623ba7e1d5a94 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=613e9d062d8c408baf7a09f021d7d5cc]), partition=
I20240501 08:12:29.872257 27871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 208771e4c9824fcf970623ba7e1d5a94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:29.873967 32565 tablet_bootstrap.cc:654] T 6c08d494131240c1ae8180f652208325 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:29.882567 32559 tablet_bootstrap.cc:492] T 6c08d494131240c1ae8180f652208325 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:29.883414 30514 tablet_service.cc:1467] Processing CreateTablet for tablet 208771e4c9824fcf970623ba7e1d5a94 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=613e9d062d8c408baf7a09f021d7d5cc]), partition=
I20240501 08:12:29.884023 30514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 208771e4c9824fcf970623ba7e1d5a94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:29.884770 28131 tablet_service.cc:1467] Processing CreateTablet for tablet 208771e4c9824fcf970623ba7e1d5a94 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=613e9d062d8c408baf7a09f021d7d5cc]), partition=
I20240501 08:12:29.885135 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 208771e4c9824fcf970623ba7e1d5a94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:29.894682 32559 tablet_bootstrap.cc:654] T 6c08d494131240c1ae8180f652208325 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:29.902022 32561 tablet_bootstrap.cc:492] T 6c08d494131240c1ae8180f652208325 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:29.903157 32570 tablet_bootstrap.cc:492] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:29.903267 32558 consensus_queue.cc:1035] T 9e22cda990da493d9aa7d9401552cb6e P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:29.905318 32570 tablet_bootstrap.cc:654] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:29.906450 32562 tablet_bootstrap.cc:492] T 28a1f9ac749143de951c1b41beec9875 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:29.906652 32562 ts_tablet_manager.cc:1367] T 28a1f9ac749143de951c1b41beec9875 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.047s user 0.008s sys 0.000s
I20240501 08:12:29.907495 32562 raft_consensus.cc:348] T 28a1f9ac749143de951c1b41beec9875 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:29.907855 32562 raft_consensus.cc:374] T 28a1f9ac749143de951c1b41beec9875 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:29.908025 32562 raft_consensus.cc:729] T 28a1f9ac749143de951c1b41beec9875 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:29.908385 32562 consensus_queue.cc:260] T 28a1f9ac749143de951c1b41beec9875 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:29.908494 32561 tablet_bootstrap.cc:654] T 6c08d494131240c1ae8180f652208325 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:29.909101 32562 ts_tablet_manager.cc:1398] T 28a1f9ac749143de951c1b41beec9875 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:29.909822 32557 tablet_bootstrap.cc:492] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:29.910207 32557 ts_tablet_manager.cc:1367] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.047s user 0.023s sys 0.000s
I20240501 08:12:29.911132 32557 raft_consensus.cc:348] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:29.911376 32557 raft_consensus.cc:374] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:29.911491 32557 raft_consensus.cc:729] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:29.911732 32557 consensus_queue.cc:260] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:29.912326 32557 ts_tablet_manager.cc:1398] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:29.912628 32557 tablet_bootstrap.cc:492] T 7affd631d1e444eaaf8a4c51267a204f P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:29.914301 32557 tablet_bootstrap.cc:654] T 7affd631d1e444eaaf8a4c51267a204f P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:29.946368 32561 tablet_bootstrap.cc:492] T 6c08d494131240c1ae8180f652208325 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:12:29.946547 32570 tablet_bootstrap.cc:492] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:12:29.946555 32561 ts_tablet_manager.cc:1367] T 6c08d494131240c1ae8180f652208325 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.009s
I20240501 08:12:29.946691 32570 ts_tablet_manager.cc:1367] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.044s user 0.000s sys 0.006s
I20240501 08:12:29.947476 32561 raft_consensus.cc:348] T 6c08d494131240c1ae8180f652208325 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:29.947765 32561 raft_consensus.cc:374] T 6c08d494131240c1ae8180f652208325 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:29.947881 32561 raft_consensus.cc:729] T 6c08d494131240c1ae8180f652208325 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:12:29.948163 32561 consensus_queue.cc:260] T 6c08d494131240c1ae8180f652208325 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:29.948763 32561 ts_tablet_manager.cc:1398] T 6c08d494131240c1ae8180f652208325 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:29.949429 32561 tablet_bootstrap.cc:492] T 208771e4c9824fcf970623ba7e1d5a94 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:29.949623 32570 raft_consensus.cc:348] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:29.949980 32570 raft_consensus.cc:374] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:29.950014 32559 tablet_bootstrap.cc:492] T 6c08d494131240c1ae8180f652208325 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:29.950174 32559 ts_tablet_manager.cc:1367] T 6c08d494131240c1ae8180f652208325 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.068s user 0.004s sys 0.000s
I20240501 08:12:29.950158 32570 raft_consensus.cc:729] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:12:29.950459 32570 consensus_queue.cc:260] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:29.950951 32559 raft_consensus.cc:348] T 6c08d494131240c1ae8180f652208325 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:29.951159 32559 raft_consensus.cc:374] T 6c08d494131240c1ae8180f652208325 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:29.951243 32559 raft_consensus.cc:729] T 6c08d494131240c1ae8180f652208325 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:29.951447 32561 tablet_bootstrap.cc:654] T 208771e4c9824fcf970623ba7e1d5a94 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:29.951448 32559 consensus_queue.cc:260] T 6c08d494131240c1ae8180f652208325 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:29.951949 32559 ts_tablet_manager.cc:1398] T 6c08d494131240c1ae8180f652208325 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:29.952234 32559 tablet_bootstrap.cc:492] T 7affd631d1e444eaaf8a4c51267a204f P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:29.952494 32570 ts_tablet_manager.cc:1398] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240501 08:12:29.962649 32559 tablet_bootstrap.cc:654] T 7affd631d1e444eaaf8a4c51267a204f P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:29.964154 32561 tablet_bootstrap.cc:492] T 208771e4c9824fcf970623ba7e1d5a94 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:12:29.964355 32561 ts_tablet_manager.cc:1367] T 208771e4c9824fcf970623ba7e1d5a94 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.003s
I20240501 08:12:29.965188 32561 raft_consensus.cc:348] T 208771e4c9824fcf970623ba7e1d5a94 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:29.965471 32561 raft_consensus.cc:374] T 208771e4c9824fcf970623ba7e1d5a94 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:29.965534 32561 raft_consensus.cc:729] T 208771e4c9824fcf970623ba7e1d5a94 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:12:29.965754 32561 consensus_queue.cc:260] T 208771e4c9824fcf970623ba7e1d5a94 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:29.966642 32561 ts_tablet_manager.cc:1398] T 208771e4c9824fcf970623ba7e1d5a94 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:29.969578 32497 raft_consensus.cc:482] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:29.969755 32497 raft_consensus.cc:504] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:29.970551 32497 leader_election.cc:288] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:29.971333 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28a1f9ac749143de951c1b41beec9875" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:29.971532 30804 raft_consensus.cc:2454] T 28a1f9ac749143de951c1b41beec9875 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 0.
I20240501 08:12:29.973450 30606 leader_election.cc:302] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:29.975243 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28a1f9ac749143de951c1b41beec9875" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
W20240501 08:12:29.975934 30607 leader_election.cc:341] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:29.980093 32497 raft_consensus.cc:2781] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:29.980217 32497 raft_consensus.cc:482] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:29.980288 32497 raft_consensus.cc:3037] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:29.983258 32497 raft_consensus.cc:504] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:29.985389 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28a1f9ac749143de951c1b41beec9875" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:29.985620 30804 raft_consensus.cc:3037] T 28a1f9ac749143de951c1b41beec9875 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:29.989187 30804 raft_consensus.cc:2454] T 28a1f9ac749143de951c1b41beec9875 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 1.
I20240501 08:12:29.990545 32559 tablet_bootstrap.cc:492] T 7affd631d1e444eaaf8a4c51267a204f P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:29.990774 32559 ts_tablet_manager.cc:1367] T 7affd631d1e444eaaf8a4c51267a204f P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.003s
I20240501 08:12:29.992920 30606 leader_election.cc:302] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:29.999215 32559 raft_consensus.cc:348] T 7affd631d1e444eaaf8a4c51267a204f P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:29.999675 32559 raft_consensus.cc:374] T 7affd631d1e444eaaf8a4c51267a204f P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:29.999848 32559 raft_consensus.cc:729] T 7affd631d1e444eaaf8a4c51267a204f P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:30.000205 32559 consensus_queue.cc:260] T 7affd631d1e444eaaf8a4c51267a204f P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:30.000989 32559 ts_tablet_manager.cc:1398] T 7affd631d1e444eaaf8a4c51267a204f P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.010s user 0.001s sys 0.002s
I20240501 08:12:30.002518 32559 tablet_bootstrap.cc:492] T 28a1f9ac749143de951c1b41beec9875 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:30.002944 32565 tablet_bootstrap.cc:492] T 6c08d494131240c1ae8180f652208325 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:30.003175 32565 ts_tablet_manager.cc:1367] T 6c08d494131240c1ae8180f652208325 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.141s user 0.004s sys 0.007s
I20240501 08:12:30.004202 32559 tablet_bootstrap.cc:654] T 28a1f9ac749143de951c1b41beec9875 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:30.004374 32565 raft_consensus.cc:348] T 6c08d494131240c1ae8180f652208325 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:30.004768 32565 raft_consensus.cc:374] T 6c08d494131240c1ae8180f652208325 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:30.004968 32565 raft_consensus.cc:729] T 6c08d494131240c1ae8180f652208325 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:30.005326 32565 consensus_queue.cc:260] T 6c08d494131240c1ae8180f652208325 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:30.006248 32565 ts_tablet_manager.cc:1398] T 6c08d494131240c1ae8180f652208325 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 08:12:30.006846 32565 tablet_bootstrap.cc:492] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:30.009531 32565 tablet_bootstrap.cc:654] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:30.013950 32497 leader_election.cc:288] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 election: Requested vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:30.014889 32497 raft_consensus.cc:2781] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:30.015102 32497 raft_consensus.cc:686] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Becoming Leader. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Running, Role: LEADER
I20240501 08:12:30.017675 32497 consensus_queue.cc:237] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:30.018697 32557 tablet_bootstrap.cc:492] T 7affd631d1e444eaaf8a4c51267a204f P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:30.020328 32557 ts_tablet_manager.cc:1367] T 7affd631d1e444eaaf8a4c51267a204f P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.108s user 0.023s sys 0.036s
I20240501 08:12:30.021589 32557 raft_consensus.cc:348] T 7affd631d1e444eaaf8a4c51267a204f P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:30.022496 32557 raft_consensus.cc:374] T 7affd631d1e444eaaf8a4c51267a204f P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:30.022581 32557 raft_consensus.cc:729] T 7affd631d1e444eaaf8a4c51267a204f P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:30.023689 32565 tablet_bootstrap.cc:492] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:30.023769 32557 consensus_queue.cc:260] T 7affd631d1e444eaaf8a4c51267a204f P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:30.023926 32565 ts_tablet_manager.cc:1367] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.006s
I20240501 08:12:30.024866 32557 ts_tablet_manager.cc:1398] T 7affd631d1e444eaaf8a4c51267a204f P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 08:12:30.026636 32565 raft_consensus.cc:348] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:30.027068 32565 raft_consensus.cc:374] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:30.027251 32565 raft_consensus.cc:729] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:30.027653 32565 consensus_queue.cc:260] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:30.028717 32565 ts_tablet_manager.cc:1398] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240501 08:12:30.029273 32565 tablet_bootstrap.cc:492] T 208771e4c9824fcf970623ba7e1d5a94 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:30.031551 32565 tablet_bootstrap.cc:654] T 208771e4c9824fcf970623ba7e1d5a94 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:30.033681 32565 tablet_bootstrap.cc:492] T 208771e4c9824fcf970623ba7e1d5a94 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:30.033967 32565 ts_tablet_manager.cc:1367] T 208771e4c9824fcf970623ba7e1d5a94 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240501 08:12:30.034770 32565 raft_consensus.cc:348] T 208771e4c9824fcf970623ba7e1d5a94 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:30.035069 32565 raft_consensus.cc:374] T 208771e4c9824fcf970623ba7e1d5a94 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:30.035207 32565 raft_consensus.cc:729] T 208771e4c9824fcf970623ba7e1d5a94 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:30.035508 32565 consensus_queue.cc:260] T 208771e4c9824fcf970623ba7e1d5a94 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:30.037842 32565 ts_tablet_manager.cc:1398] T 208771e4c9824fcf970623ba7e1d5a94 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 08:12:30.051892 32079 raft_consensus.cc:482] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:30.052181 32079 raft_consensus.cc:504] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
W20240501 08:12:30.052278 30606 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.84.135:33503, user_credentials={real_user=slave}} blocked reactor thread for 59396.9us
I20240501 08:12:30.053328 32079 leader_election.cc:288] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:30.054278 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208771e4c9824fcf970623ba7e1d5a94" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:30.054608 30536 raft_consensus.cc:2454] T 208771e4c9824fcf970623ba7e1d5a94 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:12:30.058622 27828 leader_election.cc:302] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:30.059162 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28a1f9ac749143de951c1b41beec9875" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:30.059692 32559 tablet_bootstrap.cc:492] T 28a1f9ac749143de951c1b41beec9875 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:30.059890 32559 ts_tablet_manager.cc:1367] T 28a1f9ac749143de951c1b41beec9875 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.057s user 0.016s sys 0.035s
I20240501 08:12:30.060632 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208771e4c9824fcf970623ba7e1d5a94" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:12:30.060875 28150 raft_consensus.cc:2454] T 208771e4c9824fcf970623ba7e1d5a94 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:12:30.060937 32559 raft_consensus.cc:348] T 28a1f9ac749143de951c1b41beec9875 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:30.061246 32559 raft_consensus.cc:374] T 28a1f9ac749143de951c1b41beec9875 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:30.061386 32559 raft_consensus.cc:729] T 28a1f9ac749143de951c1b41beec9875 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:30.061676 32559 consensus_queue.cc:260] T 28a1f9ac749143de951c1b41beec9875 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:30.062338 32584 raft_consensus.cc:2781] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240501 08:12:30.062985 30607 leader_election.cc:341] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:30.064069 32559 ts_tablet_manager.cc:1398] T 28a1f9ac749143de951c1b41beec9875 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 08:12:30.064224 32584 raft_consensus.cc:482] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:30.064445 32584 raft_consensus.cc:3037] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:30.075799 32584 raft_consensus.cc:504] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:30.077044 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208771e4c9824fcf970623ba7e1d5a94" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:30.077273 30536 raft_consensus.cc:3037] T 208771e4c9824fcf970623ba7e1d5a94 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:30.076822 32584 leader_election.cc:288] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Requested vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:30.079674 30536 raft_consensus.cc:2454] T 208771e4c9824fcf970623ba7e1d5a94 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:12:30.080158 27828 leader_election.cc:302] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:30.081120 32584 raft_consensus.cc:2781] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:30.081313 32584 raft_consensus.cc:686] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Becoming Leader. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Running, Role: LEADER
I20240501 08:12:30.081549 28152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "208771e4c9824fcf970623ba7e1d5a94" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:12:30.081743 28152 raft_consensus.cc:3037] T 208771e4c9824fcf970623ba7e1d5a94 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:30.081645 32584 consensus_queue.cc:237] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:30.090355 32309 raft_consensus.cc:482] T 28a1f9ac749143de951c1b41beec9875 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:30.090553 32309 raft_consensus.cc:504] T 28a1f9ac749143de951c1b41beec9875 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:30.091288 32309 leader_election.cc:288] T 28a1f9ac749143de951c1b41beec9875 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:30.092679 30670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28a1f9ac749143de951c1b41beec9875" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:30.094134 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28a1f9ac749143de951c1b41beec9875" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:30.094442 30803 raft_consensus.cc:2379] T 28a1f9ac749143de951c1b41beec9875 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a29cd590999b4f32849ff5689cd53c11 in current term 1: Already voted for candidate 9a022e1a501f4d44a0468054d6c3de47 in this term.
I20240501 08:12:30.097201 30871 leader_election.cc:302] T 28a1f9ac749143de951c1b41beec9875 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a29cd590999b4f32849ff5689cd53c11; no voters: 64998323a1f4482d8704bf20594d13a5, 9a022e1a501f4d44a0468054d6c3de47
I20240501 08:12:30.097669 32309 raft_consensus.cc:3037] T 28a1f9ac749143de951c1b41beec9875 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:30.099781 32309 raft_consensus.cc:2726] T 28a1f9ac749143de951c1b41beec9875 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:30.101007 28152 raft_consensus.cc:2454] T 208771e4c9824fcf970623ba7e1d5a94 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:12:30.111119 32500 raft_consensus.cc:482] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:30.111357 32500 raft_consensus.cc:504] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:30.112345 32500 leader_election.cc:288] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:30.113524 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7affd631d1e444eaaf8a4c51267a204f" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:30.113854 30936 raft_consensus.cc:2454] T 7affd631d1e444eaaf8a4c51267a204f P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 0.
I20240501 08:12:30.114430 30670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7affd631d1e444eaaf8a4c51267a204f" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:30.114758 30670 raft_consensus.cc:2454] T 7affd631d1e444eaaf8a4c51267a204f P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 0.
I20240501 08:12:30.115645 30470 leader_election.cc:302] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:30.116789 32444 raft_consensus.cc:2781] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:30.116894 32444 raft_consensus.cc:482] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:30.117004 32444 raft_consensus.cc:3037] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:30.118269 32309 raft_consensus.cc:482] T 6c08d494131240c1ae8180f652208325 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:30.118414 32309 raft_consensus.cc:504] T 6c08d494131240c1ae8180f652208325 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:30.119210 32309 leader_election.cc:288] T 6c08d494131240c1ae8180f652208325 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:30.119755 32444 raft_consensus.cc:504] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:30.119990 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c08d494131240c1ae8180f652208325" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:30.120187 30536 raft_consensus.cc:2454] T 6c08d494131240c1ae8180f652208325 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:30.120473 32444 leader_election.cc:288] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 election: Requested vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:30.120741 30873 leader_election.cc:302] T 6c08d494131240c1ae8180f652208325 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:30.120975 32309 raft_consensus.cc:2781] T 6c08d494131240c1ae8180f652208325 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:30.121119 32309 raft_consensus.cc:482] T 6c08d494131240c1ae8180f652208325 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:30.121227 32309 raft_consensus.cc:3037] T 6c08d494131240c1ae8180f652208325 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:30.121793 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7affd631d1e444eaaf8a4c51267a204f" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:30.121798 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c08d494131240c1ae8180f652208325" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:12:30.122022 30936 raft_consensus.cc:3037] T 7affd631d1e444eaaf8a4c51267a204f P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:30.122087 28151 raft_consensus.cc:2454] T 6c08d494131240c1ae8180f652208325 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:30.124691 32309 raft_consensus.cc:504] T 6c08d494131240c1ae8180f652208325 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:30.125339 32309 leader_election.cc:288] T 6c08d494131240c1ae8180f652208325 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Requested vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:30.125968 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c08d494131240c1ae8180f652208325" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:30.126200 30536 raft_consensus.cc:3037] T 6c08d494131240c1ae8180f652208325 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:30.126328 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c08d494131240c1ae8180f652208325" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:12:30.126541 28151 raft_consensus.cc:3037] T 6c08d494131240c1ae8180f652208325 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:30.126344 27613 catalog_manager.cc:5484] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134). New cstate: current_term: 1 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:30.127517 30936 raft_consensus.cc:2454] T 7affd631d1e444eaaf8a4c51267a204f P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 1.
I20240501 08:12:30.127887 30472 leader_election.cc:302] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:30.128144 32444 raft_consensus.cc:2781] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:30.128273 32444 raft_consensus.cc:686] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Becoming Leader. State: Replica: 31811708c6774037b82df44024b229b6, State: Running, Role: LEADER
I20240501 08:12:30.128921 30536 raft_consensus.cc:2454] T 6c08d494131240c1ae8180f652208325 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:30.129148 28151 raft_consensus.cc:2454] T 6c08d494131240c1ae8180f652208325 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:30.129407 30873 leader_election.cc:302] T 6c08d494131240c1ae8180f652208325 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:30.128616 32444 consensus_queue.cc:237] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:30.129801 32214 raft_consensus.cc:2781] T 6c08d494131240c1ae8180f652208325 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:30.129997 32214 raft_consensus.cc:686] T 6c08d494131240c1ae8180f652208325 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Becoming Leader. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:12:30.130244 32214 consensus_queue.cc:237] T 6c08d494131240c1ae8180f652208325 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:30.133101 30670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7affd631d1e444eaaf8a4c51267a204f" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:30.133373 30670 raft_consensus.cc:3037] T 7affd631d1e444eaaf8a4c51267a204f P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:30.135972 30670 raft_consensus.cc:2454] T 7affd631d1e444eaaf8a4c51267a204f P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 1.
I20240501 08:12:30.137951 27620 catalog_manager.cc:5484] T 6c08d494131240c1ae8180f652208325 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: term changed from 0 to 1, leader changed from <none> to a29cd590999b4f32849ff5689cd53c11 (127.21.84.136). New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:30.140616 27613 catalog_manager.cc:5484] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af reported cstate change: term changed from 0 to 1, leader changed from <none> to f1525cbc82a24d13b65f136695c903af (127.21.84.130). New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:30.148082 27613 catalog_manager.cc:5484] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31811708c6774037b82df44024b229b6 (127.21.84.133). New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:30.272022 27613 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:30.278656 30536 raft_consensus.cc:1261] T 208771e4c9824fcf970623ba7e1d5a94 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:30.278925 28151 raft_consensus.cc:1261] T 208771e4c9824fcf970623ba7e1d5a94 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:30.279166 32584 consensus_queue.cc:1035] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:30.279498 32585 consensus_queue.cc:1035] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:30.286923 27616 catalog_manager.cc:5576] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323) does not have the latest schema for tablet 208771e4c9824fcf970623ba7e1d5a94 (table rebalancer_dynamic_table [id=613e9d062d8c408baf7a09f021d7d5cc]). Expected version 1 got 0
I20240501 08:12:30.289846 32593 tablet.cc:1722] T 208771e4c9824fcf970623ba7e1d5a94 P 91d11eae5ef6432ca86eb8ed446b6ea6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:30.292749 32593 tablet.cc:1620] T 208771e4c9824fcf970623ba7e1d5a94 P 91d11eae5ef6432ca86eb8ed446b6ea6: MemRowSet was empty: no flush needed.
I20240501 08:12:30.295209 27613 catalog_manager.cc:5576] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293) does not have the latest schema for tablet 208771e4c9824fcf970623ba7e1d5a94 (table rebalancer_dynamic_table [id=613e9d062d8c408baf7a09f021d7d5cc]). Expected version 1 got 0
I20240501 08:12:30.295805 32592 tablet.cc:1722] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:30.298074 32594 tablet.cc:1722] T 208771e4c9824fcf970623ba7e1d5a94 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:30.299618 32594 tablet.cc:1620] T 208771e4c9824fcf970623ba7e1d5a94 P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:30.299626 32592 tablet.cc:1620] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af: MemRowSet was empty: no flush needed.
I20240501 08:12:30.300716 27620 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=613e9d062d8c408baf7a09f021d7d5cc] alter complete (version 1)
I20240501 08:12:30.407433 27620 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:30.416468 32593 tablet.cc:1722] T 208771e4c9824fcf970623ba7e1d5a94 P 91d11eae5ef6432ca86eb8ed446b6ea6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:30.416985 32593 tablet.cc:1620] T 208771e4c9824fcf970623ba7e1d5a94 P 91d11eae5ef6432ca86eb8ed446b6ea6: MemRowSet was empty: no flush needed.
I20240501 08:12:30.417058 32592 tablet.cc:1722] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:30.417465 32592 tablet.cc:1620] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af: MemRowSet was empty: no flush needed.
I20240501 08:12:30.417414 32594 tablet.cc:1722] T 208771e4c9824fcf970623ba7e1d5a94 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:30.418040 32594 tablet.cc:1620] T 208771e4c9824fcf970623ba7e1d5a94 P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:30.425410 32497 consensus_queue.cc:1035] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:30.428169 27620 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=613e9d062d8c408baf7a09f021d7d5cc] alter complete (version 2)
I20240501 08:12:30.437875 32497 consensus_queue.cc:1035] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:30.528951 32444 consensus_queue.cc:1035] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:30.535122 32500 consensus_queue.cc:1035] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:30.536507 27613 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:30.546129 32592 tablet.cc:1722] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:30.547142 32594 tablet.cc:1722] T 208771e4c9824fcf970623ba7e1d5a94 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:30.556964 32593 tablet.cc:1722] T 208771e4c9824fcf970623ba7e1d5a94 P 91d11eae5ef6432ca86eb8ed446b6ea6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:30.559188 27616 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table_new_name [id=613e9d062d8c408baf7a09f021d7d5cc] alter complete (version 3)
I20240501 08:12:30.619989 32309 consensus_queue.cc:1035] T 6c08d494131240c1ae8180f652208325 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:30.624117 32563 consensus_queue.cc:1035] T 6c08d494131240c1ae8180f652208325 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:30.661227 27620 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:30.661492 27620 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:30.664201 27620 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 208771e4c9824fcf970623ba7e1d5a94
I20240501 08:12:30.664963 27871 tablet_service.cc:1514] Processing DeleteTablet for tablet 208771e4c9824fcf970623ba7e1d5a94 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:30 UTC) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:30.665273 30514 tablet_service.cc:1514] Processing DeleteTablet for tablet 208771e4c9824fcf970623ba7e1d5a94 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:30 UTC) from {username='slave'} at 127.0.0.1:45078
I20240501 08:12:30.665395 28131 tablet_service.cc:1514] Processing DeleteTablet for tablet 208771e4c9824fcf970623ba7e1d5a94 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:30 UTC) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:30.665596 32607 tablet_replica.cc:330] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:30.665807 32607 raft_consensus.cc:2229] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:30.666209 32607 raft_consensus.cc:2258] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:30.667145 32607 ts_tablet_manager.cc:1875] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:30.671770 32607 ts_tablet_manager.cc:1888] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:30.671888 32607 log.cc:1196] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/208771e4c9824fcf970623ba7e1d5a94
I20240501 08:12:30.672317 32607 ts_tablet_manager.cc:1909] T 208771e4c9824fcf970623ba7e1d5a94 P f1525cbc82a24d13b65f136695c903af: Deleting consensus metadata
I20240501 08:12:30.673146 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet 208771e4c9824fcf970623ba7e1d5a94 (table rebalancer_dynamic_table_new_name [id=613e9d062d8c408baf7a09f021d7d5cc]) successfully deleted
I20240501 08:12:30.675686 32609 tablet_replica.cc:330] T 208771e4c9824fcf970623ba7e1d5a94 P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:30.675807 32608 tablet_replica.cc:330] T 208771e4c9824fcf970623ba7e1d5a94 P 31811708c6774037b82df44024b229b6: stopping tablet replica
I20240501 08:12:30.675907 32609 raft_consensus.cc:2229] T 208771e4c9824fcf970623ba7e1d5a94 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:30.676012 32608 raft_consensus.cc:2229] T 208771e4c9824fcf970623ba7e1d5a94 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:30.676107 32609 raft_consensus.cc:2258] T 208771e4c9824fcf970623ba7e1d5a94 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:30.676203 32608 raft_consensus.cc:2258] T 208771e4c9824fcf970623ba7e1d5a94 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:30.676932 32609 ts_tablet_manager.cc:1875] T 208771e4c9824fcf970623ba7e1d5a94 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:30.678439 32608 ts_tablet_manager.cc:1875] T 208771e4c9824fcf970623ba7e1d5a94 P 31811708c6774037b82df44024b229b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:30.683182 32608 ts_tablet_manager.cc:1888] T 208771e4c9824fcf970623ba7e1d5a94 P 31811708c6774037b82df44024b229b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:30.683277 32608 log.cc:1196] T 208771e4c9824fcf970623ba7e1d5a94 P 31811708c6774037b82df44024b229b6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/wals/208771e4c9824fcf970623ba7e1d5a94
I20240501 08:12:30.683593 32608 ts_tablet_manager.cc:1909] T 208771e4c9824fcf970623ba7e1d5a94 P 31811708c6774037b82df44024b229b6: Deleting consensus metadata
I20240501 08:12:30.683802 32609 ts_tablet_manager.cc:1888] T 208771e4c9824fcf970623ba7e1d5a94 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:30.683964 32609 log.cc:1196] T 208771e4c9824fcf970623ba7e1d5a94 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/208771e4c9824fcf970623ba7e1d5a94
I20240501 08:12:30.684429 27607 catalog_manager.cc:4830] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): tablet 208771e4c9824fcf970623ba7e1d5a94 (table rebalancer_dynamic_table_new_name [id=613e9d062d8c408baf7a09f021d7d5cc]) successfully deleted
I20240501 08:12:30.684485 32609 ts_tablet_manager.cc:1909] T 208771e4c9824fcf970623ba7e1d5a94 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting consensus metadata
I20240501 08:12:30.685508 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet 208771e4c9824fcf970623ba7e1d5a94 (table rebalancer_dynamic_table_new_name [id=613e9d062d8c408baf7a09f021d7d5cc]) successfully deleted
I20240501 08:12:30.707516 27620 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_extra_table_56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:30.716547 30652 tablet_service.cc:1467] Processing CreateTablet for tablet 4973818ff02b4b7eba922b7fe9193ca6 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=2e283cf00add44fc94e82872e4b0aab8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:30.716708 27741 tablet_service.cc:1467] Processing CreateTablet for tablet 4973818ff02b4b7eba922b7fe9193ca6 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=2e283cf00add44fc94e82872e4b0aab8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:30.716956 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4973818ff02b4b7eba922b7fe9193ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:30.717105 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4973818ff02b4b7eba922b7fe9193ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:30.721633 28002 tablet_service.cc:1467] Processing CreateTablet for tablet 4973818ff02b4b7eba922b7fe9193ca6 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=2e283cf00add44fc94e82872e4b0aab8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:30.722132 32610 tablet_bootstrap.cc:492] T 4973818ff02b4b7eba922b7fe9193ca6 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:30.722190 28002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4973818ff02b4b7eba922b7fe9193ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:30.724073 32610 tablet_bootstrap.cc:654] T 4973818ff02b4b7eba922b7fe9193ca6 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:30.725421 28131 tablet_service.cc:1467] Processing CreateTablet for tablet d020e1225d5d447f993f920569a05f3d (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=2e283cf00add44fc94e82872e4b0aab8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:30.725814 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d020e1225d5d447f993f920569a05f3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:30.726114 27871 tablet_service.cc:1467] Processing CreateTablet for tablet d020e1225d5d447f993f920569a05f3d (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=2e283cf00add44fc94e82872e4b0aab8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:30.726487 27871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d020e1225d5d447f993f920569a05f3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:30.729163 27872 tablet_service.cc:1467] Processing CreateTablet for tablet b0906abad5c14912b94e9c017d2ecf8d (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=2e283cf00add44fc94e82872e4b0aab8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:30.729568 27872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0906abad5c14912b94e9c017d2ecf8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:30.729686 27739 tablet_service.cc:1467] Processing CreateTablet for tablet b0906abad5c14912b94e9c017d2ecf8d (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=2e283cf00add44fc94e82872e4b0aab8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:30.730337 27739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0906abad5c14912b94e9c017d2ecf8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:30.735417 32611 tablet_bootstrap.cc:492] T 4973818ff02b4b7eba922b7fe9193ca6 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:30.737630 32611 tablet_bootstrap.cc:654] T 4973818ff02b4b7eba922b7fe9193ca6 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:30.741515 32615 tablet_bootstrap.cc:492] T 4973818ff02b4b7eba922b7fe9193ca6 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:30.743633 32615 tablet_bootstrap.cc:654] T 4973818ff02b4b7eba922b7fe9193ca6 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:30.744426 32612 tablet_bootstrap.cc:492] T d020e1225d5d447f993f920569a05f3d P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:30.744554 32611 tablet_bootstrap.cc:492] T 4973818ff02b4b7eba922b7fe9193ca6 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:12:30.744694 32611 ts_tablet_manager.cc:1367] T 4973818ff02b4b7eba922b7fe9193ca6 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240501 08:12:30.745424 32611 raft_consensus.cc:348] T 4973818ff02b4b7eba922b7fe9193ca6 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:30.745604 32611 raft_consensus.cc:374] T 4973818ff02b4b7eba922b7fe9193ca6 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:30.745682 32611 raft_consensus.cc:729] T 4973818ff02b4b7eba922b7fe9193ca6 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:12:30.745900 32611 consensus_queue.cc:260] T 4973818ff02b4b7eba922b7fe9193ca6 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:30.746234 30784 tablet_service.cc:1467] Processing CreateTablet for tablet b0906abad5c14912b94e9c017d2ecf8d (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=2e283cf00add44fc94e82872e4b0aab8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:30.746696 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0906abad5c14912b94e9c017d2ecf8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:30.746822 32611 ts_tablet_manager.cc:1398] T 4973818ff02b4b7eba922b7fe9193ca6 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:30.747104 32611 tablet_bootstrap.cc:492] T b0906abad5c14912b94e9c017d2ecf8d P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:30.746265 32612 tablet_bootstrap.cc:654] T d020e1225d5d447f993f920569a05f3d P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:30.748560 32611 tablet_bootstrap.cc:654] T b0906abad5c14912b94e9c017d2ecf8d P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:30.759946 32615 tablet_bootstrap.cc:492] T 4973818ff02b4b7eba922b7fe9193ca6 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:12:30.760267 32615 ts_tablet_manager.cc:1367] T 4973818ff02b4b7eba922b7fe9193ca6 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.005s
I20240501 08:12:30.762027 32619 tablet_bootstrap.cc:492] T b0906abad5c14912b94e9c017d2ecf8d P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:30.761898 32615 raft_consensus.cc:348] T 4973818ff02b4b7eba922b7fe9193ca6 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:30.762275 32615 raft_consensus.cc:374] T 4973818ff02b4b7eba922b7fe9193ca6 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:30.763289 32611 tablet_bootstrap.cc:492] T b0906abad5c14912b94e9c017d2ecf8d P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:12:30.764169 32619 tablet_bootstrap.cc:654] T b0906abad5c14912b94e9c017d2ecf8d P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:30.765079 32611 ts_tablet_manager.cc:1367] T b0906abad5c14912b94e9c017d2ecf8d P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.000s
I20240501 08:12:30.766594 32610 tablet_bootstrap.cc:492] T 4973818ff02b4b7eba922b7fe9193ca6 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:30.766758 32610 ts_tablet_manager.cc:1367] T 4973818ff02b4b7eba922b7fe9193ca6 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.004s
I20240501 08:12:30.767313 32611 raft_consensus.cc:348] T b0906abad5c14912b94e9c017d2ecf8d P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:30.768185 32610 raft_consensus.cc:348] T 4973818ff02b4b7eba922b7fe9193ca6 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:30.768368 32610 raft_consensus.cc:374] T 4973818ff02b4b7eba922b7fe9193ca6 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:30.768455 32610 raft_consensus.cc:729] T 4973818ff02b4b7eba922b7fe9193ca6 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:30.768427 30514 tablet_service.cc:1467] Processing CreateTablet for tablet d020e1225d5d447f993f920569a05f3d (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=2e283cf00add44fc94e82872e4b0aab8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:30.768656 32610 consensus_queue.cc:260] T 4973818ff02b4b7eba922b7fe9193ca6 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:30.768894 30514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d020e1225d5d447f993f920569a05f3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:30.769207 32610 ts_tablet_manager.cc:1398] T 4973818ff02b4b7eba922b7fe9193ca6 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:30.769574 27613 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:30.770632 32611 raft_consensus.cc:374] T b0906abad5c14912b94e9c017d2ecf8d P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:30.770748 32611 raft_consensus.cc:729] T b0906abad5c14912b94e9c017d2ecf8d P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:12:30.771104 32611 consensus_queue.cc:260] T b0906abad5c14912b94e9c017d2ecf8d P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:30.771719 32615 raft_consensus.cc:729] T 4973818ff02b4b7eba922b7fe9193ca6 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:12:30.772059 32615 consensus_queue.cc:260] T 4973818ff02b4b7eba922b7fe9193ca6 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:30.772796 32611 ts_tablet_manager.cc:1398] T b0906abad5c14912b94e9c017d2ecf8d P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:12:30.773432 32615 ts_tablet_manager.cc:1398] T 4973818ff02b4b7eba922b7fe9193ca6 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240501 08:12:30.789180 32578 raft_consensus.cc:482] T 4973818ff02b4b7eba922b7fe9193ca6 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:30.789431 32578 raft_consensus.cc:504] T 4973818ff02b4b7eba922b7fe9193ca6 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:30.790452 32578 leader_election.cc:288] T 4973818ff02b4b7eba922b7fe9193ca6 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:30.790633 28002 tablet_service.cc:1467] Processing CreateTablet for tablet 12f63fbe69e144a8ae21d881c122ad7a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5865465fbe8b4b5f8004e22f8f68f462]), partition=
I20240501 08:12:30.791177 28002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12f63fbe69e144a8ae21d881c122ad7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:30.791301 28131 tablet_service.cc:1467] Processing CreateTablet for tablet 12f63fbe69e144a8ae21d881c122ad7a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5865465fbe8b4b5f8004e22f8f68f462]), partition=
I20240501 08:12:30.791662 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12f63fbe69e144a8ae21d881c122ad7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:30.791831 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4973818ff02b4b7eba922b7fe9193ca6" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:12:30.792058 27759 raft_consensus.cc:2454] T 4973818ff02b4b7eba922b7fe9193ca6 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 0.
I20240501 08:12:30.792516 30607 leader_election.cc:302] T 4973818ff02b4b7eba922b7fe9193ca6 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:30.794257 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4973818ff02b4b7eba922b7fe9193ca6" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:12:30.794479 28022 raft_consensus.cc:2454] T 4973818ff02b4b7eba922b7fe9193ca6 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 0.
I20240501 08:12:30.794493 27870 tablet_service.cc:1467] Processing CreateTablet for tablet 12f63fbe69e144a8ae21d881c122ad7a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5865465fbe8b4b5f8004e22f8f68f462]), partition=
I20240501 08:12:30.794937 27870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12f63fbe69e144a8ae21d881c122ad7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:30.797235 32578 raft_consensus.cc:2781] T 4973818ff02b4b7eba922b7fe9193ca6 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:30.797351 32578 raft_consensus.cc:482] T 4973818ff02b4b7eba922b7fe9193ca6 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:30.797426 32578 raft_consensus.cc:3037] T 4973818ff02b4b7eba922b7fe9193ca6 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:30.804643 32578 raft_consensus.cc:504] T 4973818ff02b4b7eba922b7fe9193ca6 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:30.809614 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4973818ff02b4b7eba922b7fe9193ca6" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:12:30.809870 28022 raft_consensus.cc:3037] T 4973818ff02b4b7eba922b7fe9193ca6 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:30.815246 32615 tablet_bootstrap.cc:492] T 12f63fbe69e144a8ae21d881c122ad7a P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:30.815515 28022 raft_consensus.cc:2454] T 4973818ff02b4b7eba922b7fe9193ca6 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 1.
I20240501 08:12:30.817118 30607 leader_election.cc:302] T 4973818ff02b4b7eba922b7fe9193ca6 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a022e1a501f4d44a0468054d6c3de47, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:12:30.817468 32580 raft_consensus.cc:2781] T 4973818ff02b4b7eba922b7fe9193ca6 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:30.817606 32580 raft_consensus.cc:686] T 4973818ff02b4b7eba922b7fe9193ca6 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Becoming Leader. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Running, Role: LEADER
I20240501 08:12:30.818013 32615 tablet_bootstrap.cc:654] T 12f63fbe69e144a8ae21d881c122ad7a P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:30.817981 32580 consensus_queue.cc:237] T 4973818ff02b4b7eba922b7fe9193ca6 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:12:30.826453 32621 tablet_bootstrap.cc:492] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:30.829026 32621 tablet_bootstrap.cc:654] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:30.836313 32621 tablet_bootstrap.cc:492] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:12:30.836474 32621 ts_tablet_manager.cc:1367] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240501 08:12:30.839650 32621 raft_consensus.cc:348] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:30.839890 32621 raft_consensus.cc:374] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:30.840005 32621 raft_consensus.cc:729] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:12:30.842144 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4973818ff02b4b7eba922b7fe9193ca6" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:12:30.842484 27759 raft_consensus.cc:3037] T 4973818ff02b4b7eba922b7fe9193ca6 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:30.842875 32621 consensus_queue.cc:260] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:30.844283 32621 ts_tablet_manager.cc:1398] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 08:12:30.862234 32578 leader_election.cc:288] T 4973818ff02b4b7eba922b7fe9193ca6 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 election: Requested vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:30.879142 32621 tablet_bootstrap.cc:492] T b0906abad5c14912b94e9c017d2ecf8d P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:30.879983 27759 raft_consensus.cc:2454] T 4973818ff02b4b7eba922b7fe9193ca6 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 1.
I20240501 08:12:30.881420 32621 tablet_bootstrap.cc:654] T b0906abad5c14912b94e9c017d2ecf8d P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:30.900652 32621 tablet_bootstrap.cc:492] T b0906abad5c14912b94e9c017d2ecf8d P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:12:30.900933 32621 ts_tablet_manager.cc:1367] T b0906abad5c14912b94e9c017d2ecf8d P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.005s
I20240501 08:12:30.901753 32621 raft_consensus.cc:348] T b0906abad5c14912b94e9c017d2ecf8d P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:30.902169 32621 raft_consensus.cc:374] T b0906abad5c14912b94e9c017d2ecf8d P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:30.902294 32621 raft_consensus.cc:729] T b0906abad5c14912b94e9c017d2ecf8d P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:12:30.902585 32621 consensus_queue.cc:260] T b0906abad5c14912b94e9c017d2ecf8d P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:30.903353 32621 ts_tablet_manager.cc:1398] T b0906abad5c14912b94e9c017d2ecf8d P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:30.906757 32621 tablet_bootstrap.cc:492] T d020e1225d5d447f993f920569a05f3d P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:30.908437 32621 tablet_bootstrap.cc:654] T d020e1225d5d447f993f920569a05f3d P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:30.911451 32619 tablet_bootstrap.cc:492] T b0906abad5c14912b94e9c017d2ecf8d P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:30.911661 32619 ts_tablet_manager.cc:1367] T b0906abad5c14912b94e9c017d2ecf8d P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.150s user 0.029s sys 0.080s
I20240501 08:12:30.911906 32612 tablet_bootstrap.cc:492] T d020e1225d5d447f993f920569a05f3d P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:12:30.912094 32612 ts_tablet_manager.cc:1367] T d020e1225d5d447f993f920569a05f3d P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.168s user 0.036s sys 0.033s
I20240501 08:12:30.912503 32619 raft_consensus.cc:348] T b0906abad5c14912b94e9c017d2ecf8d P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:30.912787 32619 raft_consensus.cc:374] T b0906abad5c14912b94e9c017d2ecf8d P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:30.912932 32619 raft_consensus.cc:729] T b0906abad5c14912b94e9c017d2ecf8d P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:30.913167 32612 raft_consensus.cc:348] T d020e1225d5d447f993f920569a05f3d P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:30.913436 32612 raft_consensus.cc:374] T d020e1225d5d447f993f920569a05f3d P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:30.913352 32619 consensus_queue.cc:260] T b0906abad5c14912b94e9c017d2ecf8d P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:30.913542 32612 raft_consensus.cc:729] T d020e1225d5d447f993f920569a05f3d P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:12:30.913842 32612 consensus_queue.cc:260] T d020e1225d5d447f993f920569a05f3d P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:30.914110 32619 ts_tablet_manager.cc:1398] T b0906abad5c14912b94e9c017d2ecf8d P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:30.915819 32612 ts_tablet_manager.cc:1398] T d020e1225d5d447f993f920569a05f3d P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 08:12:30.916080 32612 tablet_bootstrap.cc:492] T 12f63fbe69e144a8ae21d881c122ad7a P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:30.918623 32612 tablet_bootstrap.cc:654] T 12f63fbe69e144a8ae21d881c122ad7a P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:30.920653 32621 tablet_bootstrap.cc:492] T d020e1225d5d447f993f920569a05f3d P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:12:30.921262 32621 ts_tablet_manager.cc:1367] T d020e1225d5d447f993f920569a05f3d P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.005s
I20240501 08:12:30.922364 32621 raft_consensus.cc:348] T d020e1225d5d447f993f920569a05f3d P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:30.923537 32621 raft_consensus.cc:374] T d020e1225d5d447f993f920569a05f3d P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:30.923800 32621 raft_consensus.cc:729] T d020e1225d5d447f993f920569a05f3d P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:12:30.924221 32621 consensus_queue.cc:260] T d020e1225d5d447f993f920569a05f3d P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:30.927269 32621 ts_tablet_manager.cc:1398] T d020e1225d5d447f993f920569a05f3d P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:12:30.943949 32612 tablet_bootstrap.cc:492] T 12f63fbe69e144a8ae21d881c122ad7a P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:12:30.944162 32612 ts_tablet_manager.cc:1367] T 12f63fbe69e144a8ae21d881c122ad7a P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.005s
I20240501 08:12:30.945082 32612 raft_consensus.cc:348] T 12f63fbe69e144a8ae21d881c122ad7a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:30.945361 32612 raft_consensus.cc:374] T 12f63fbe69e144a8ae21d881c122ad7a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:30.945411 32612 raft_consensus.cc:729] T 12f63fbe69e144a8ae21d881c122ad7a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:12:30.945658 32612 consensus_queue.cc:260] T 12f63fbe69e144a8ae21d881c122ad7a P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:30.946238 32612 ts_tablet_manager.cc:1398] T 12f63fbe69e144a8ae21d881c122ad7a P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:12:30.948930 32628 tablet_bootstrap.cc:492] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:30.950960 32628 tablet_bootstrap.cc:654] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:30.955476 32584 raft_consensus.cc:482] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:30.955658 32584 raft_consensus.cc:504] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:30.956383 32584 leader_election.cc:288] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:30.956670 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f63fbe69e144a8ae21d881c122ad7a" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:12:30.956852 28151 raft_consensus.cc:2454] T 12f63fbe69e144a8ae21d881c122ad7a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:12:30.957580 27825 leader_election.cc:302] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:30.959184 32079 raft_consensus.cc:2781] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:30.959280 32079 raft_consensus.cc:482] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:30.959375 32079 raft_consensus.cc:3037] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:30.964071 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f63fbe69e144a8ae21d881c122ad7a" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
W20240501 08:12:30.965453 27827 leader_election.cc:341] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:30.966961 27616 catalog_manager.cc:5484] T 4973818ff02b4b7eba922b7fe9193ca6 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134). New cstate: current_term: 1 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:30.973444 32079 raft_consensus.cc:504] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:30.975271 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f63fbe69e144a8ae21d881c122ad7a" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:12:30.975466 28151 raft_consensus.cc:3037] T 12f63fbe69e144a8ae21d881c122ad7a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:30.980342 28151 raft_consensus.cc:2454] T 12f63fbe69e144a8ae21d881c122ad7a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:12:30.981022 27825 leader_election.cc:302] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:30.981555 32584 raft_consensus.cc:2781] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:30.981698 32584 raft_consensus.cc:686] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Becoming Leader. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Running, Role: LEADER
I20240501 08:12:30.986301 32584 consensus_queue.cc:237] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:30.988081 32079 leader_election.cc:288] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Requested vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:30.988875 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f63fbe69e144a8ae21d881c122ad7a" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
W20240501 08:12:30.989578 27827 leader_election.cc:341] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:30.992456 32628 tablet_bootstrap.cc:492] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:30.992705 32628 ts_tablet_manager.cc:1367] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.044s user 0.000s sys 0.010s
I20240501 08:12:30.993752 32628 raft_consensus.cc:348] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:30.994174 32628 raft_consensus.cc:374] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:30.994344 32628 raft_consensus.cc:729] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:30.994695 32628 consensus_queue.cc:260] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:30.996827 32628 ts_tablet_manager.cc:1398] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 08:12:31.000576 27618 catalog_manager.cc:5484] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af reported cstate change: term changed from 0 to 1, leader changed from <none> to f1525cbc82a24d13b65f136695c903af (127.21.84.130). New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:31.007858 32615 tablet_bootstrap.cc:492] T 12f63fbe69e144a8ae21d881c122ad7a P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:12:31.008023 32615 ts_tablet_manager.cc:1367] T 12f63fbe69e144a8ae21d881c122ad7a P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.193s user 0.064s sys 0.064s
I20240501 08:12:31.008788 32615 raft_consensus.cc:348] T 12f63fbe69e144a8ae21d881c122ad7a P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:31.009006 32615 raft_consensus.cc:374] T 12f63fbe69e144a8ae21d881c122ad7a P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:31.009076 32615 raft_consensus.cc:729] T 12f63fbe69e144a8ae21d881c122ad7a P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:12:31.009369 32615 consensus_queue.cc:260] T 12f63fbe69e144a8ae21d881c122ad7a P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:31.009979 32615 ts_tablet_manager.cc:1398] T 12f63fbe69e144a8ae21d881c122ad7a P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:31.029521 32564 raft_consensus.cc:482] T b0906abad5c14912b94e9c017d2ecf8d P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:31.029706 32500 raft_consensus.cc:482] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:31.029755 32564 raft_consensus.cc:504] T b0906abad5c14912b94e9c017d2ecf8d P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:31.029875 32500 raft_consensus.cc:504] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:31.030601 32564 leader_election.cc:288] T b0906abad5c14912b94e9c017d2ecf8d P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:31.030589 32500 leader_election.cc:288] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:12:31.031250 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0906abad5c14912b94e9c017d2ecf8d" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:31.031314 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d020e1225d5d447f993f920569a05f3d" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:12:31.031499 30803 raft_consensus.cc:2454] T b0906abad5c14912b94e9c017d2ecf8d P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:12:31.031545 27890 raft_consensus.cc:2454] T d020e1225d5d447f993f920569a05f3d P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 0.
I20240501 08:12:31.031608 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d020e1225d5d447f993f920569a05f3d" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:12:31.031801 28151 raft_consensus.cc:2454] T d020e1225d5d447f993f920569a05f3d P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 0.
I20240501 08:12:31.032016 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0906abad5c14912b94e9c017d2ecf8d" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:12:31.032243 27889 raft_consensus.cc:2454] T b0906abad5c14912b94e9c017d2ecf8d P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:12:31.032287 30470 leader_election.cc:302] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:12:31.032728 27697 leader_election.cc:302] T b0906abad5c14912b94e9c017d2ecf8d P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:31.033203 32500 raft_consensus.cc:2781] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:31.033324 32500 raft_consensus.cc:482] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:31.033433 32500 raft_consensus.cc:3037] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:31.034055 32564 raft_consensus.cc:2781] T b0906abad5c14912b94e9c017d2ecf8d P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:31.034204 32564 raft_consensus.cc:482] T b0906abad5c14912b94e9c017d2ecf8d P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:31.034283 32564 raft_consensus.cc:3037] T b0906abad5c14912b94e9c017d2ecf8d P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:31.036377 32564 raft_consensus.cc:504] T b0906abad5c14912b94e9c017d2ecf8d P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:31.036548 32500 raft_consensus.cc:504] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:31.037653 32500 leader_election.cc:288] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 election: Requested vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:12:31.037690 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0906abad5c14912b94e9c017d2ecf8d" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:12:31.037772 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d020e1225d5d447f993f920569a05f3d" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:12:31.037858 27889 raft_consensus.cc:3037] T b0906abad5c14912b94e9c017d2ecf8d P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:31.037986 28151 raft_consensus.cc:3037] T d020e1225d5d447f993f920569a05f3d P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:31.038233 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d020e1225d5d447f993f920569a05f3d" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:12:31.038409 27890 raft_consensus.cc:3037] T d020e1225d5d447f993f920569a05f3d P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:31.040603 27889 raft_consensus.cc:2454] T b0906abad5c14912b94e9c017d2ecf8d P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:12:31.040629 28151 raft_consensus.cc:2454] T d020e1225d5d447f993f920569a05f3d P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 1.
I20240501 08:12:31.041030 30470 leader_election.cc:302] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:12:31.041098 27890 raft_consensus.cc:2454] T d020e1225d5d447f993f920569a05f3d P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 1.
I20240501 08:12:31.041294 32500 raft_consensus.cc:2781] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:31.041464 32500 raft_consensus.cc:686] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Becoming Leader. State: Replica: 31811708c6774037b82df44024b229b6, State: Running, Role: LEADER
I20240501 08:12:31.041723 32500 consensus_queue.cc:237] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:31.042052 32564 leader_election.cc:288] T b0906abad5c14912b94e9c017d2ecf8d P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Requested vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:31.043365 27697 leader_election.cc:302] T b0906abad5c14912b94e9c017d2ecf8d P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:31.044003 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0906abad5c14912b94e9c017d2ecf8d" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:31.044241 30803 raft_consensus.cc:3037] T b0906abad5c14912b94e9c017d2ecf8d P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:31.046775 32564 raft_consensus.cc:2781] T b0906abad5c14912b94e9c017d2ecf8d P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:31.046944 32564 raft_consensus.cc:686] T b0906abad5c14912b94e9c017d2ecf8d P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:12:31.047266 30803 raft_consensus.cc:2454] T b0906abad5c14912b94e9c017d2ecf8d P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:12:31.047242 32564 consensus_queue.cc:237] T b0906abad5c14912b94e9c017d2ecf8d P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:31.061304 27620 catalog_manager.cc:5484] T b0906abad5c14912b94e9c017d2ecf8d P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:31.069967 27620 catalog_manager.cc:5484] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31811708c6774037b82df44024b229b6 (127.21.84.133). New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:31.138206 27620 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:31.143923 28022 raft_consensus.cc:3037] T 12f63fbe69e144a8ae21d881c122ad7a P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:31.144048 28152 raft_consensus.cc:1261] T 12f63fbe69e144a8ae21d881c122ad7a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:31.144645 32584 consensus_queue.cc:1035] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:31.146612 28022 raft_consensus.cc:1261] T 12f63fbe69e144a8ae21d881c122ad7a P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:31.147119 32584 consensus_queue.cc:1035] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:31.149833 27620 catalog_manager.cc:5576] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323) does not have the latest schema for tablet 12f63fbe69e144a8ae21d881c122ad7a (table rebalancer_dynamic_table [id=5865465fbe8b4b5f8004e22f8f68f462]). Expected version 1 got 0
I20240501 08:12:31.153949 32638 tablet.cc:1722] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:31.154700 32638 tablet.cc:1620] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af: MemRowSet was empty: no flush needed.
I20240501 08:12:31.155457 27620 catalog_manager.cc:5576] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323) does not have the latest schema for tablet 12f63fbe69e144a8ae21d881c122ad7a (table rebalancer_dynamic_table [id=5865465fbe8b4b5f8004e22f8f68f462]). Expected version 1 got 0
I20240501 08:12:31.158778 27618 catalog_manager.cc:5576] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503) does not have the latest schema for tablet 12f63fbe69e144a8ae21d881c122ad7a (table rebalancer_dynamic_table [id=5865465fbe8b4b5f8004e22f8f68f462]). Expected version 1 got 0
I20240501 08:12:31.161130 32640 tablet.cc:1722] T 12f63fbe69e144a8ae21d881c122ad7a P 91d11eae5ef6432ca86eb8ed446b6ea6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:31.161582 32640 tablet.cc:1620] T 12f63fbe69e144a8ae21d881c122ad7a P 91d11eae5ef6432ca86eb8ed446b6ea6: MemRowSet was empty: no flush needed.
I20240501 08:12:31.166085 27616 catalog_manager.cc:5576] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305) does not have the latest schema for tablet 12f63fbe69e144a8ae21d881c122ad7a (table rebalancer_dynamic_table [id=5865465fbe8b4b5f8004e22f8f68f462]). Expected version 1 got 0
I20240501 08:12:31.166352 27620 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=5865465fbe8b4b5f8004e22f8f68f462] alter complete (version 1)
I20240501 08:12:31.169337 32643 tablet.cc:1722] T 12f63fbe69e144a8ae21d881c122ad7a P aad81047009a4ffcbc450b60030583b5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:31.170179 32643 tablet.cc:1620] T 12f63fbe69e144a8ae21d881c122ad7a P aad81047009a4ffcbc450b60030583b5: MemRowSet was empty: no flush needed.
I20240501 08:12:31.172109 27618 catalog_manager.cc:5576] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305) does not have the latest schema for tablet 12f63fbe69e144a8ae21d881c122ad7a (table rebalancer_dynamic_table [id=5865465fbe8b4b5f8004e22f8f68f462]). Expected version 1 got 0
I20240501 08:12:31.203891 32497 consensus_queue.cc:1035] T 4973818ff02b4b7eba922b7fe9193ca6 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:31.212743 32497 consensus_queue.cc:1035] T 4973818ff02b4b7eba922b7fe9193ca6 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:31.279675 27616 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:31.286916 32638 tablet.cc:1722] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:31.287467 32638 tablet.cc:1620] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af: MemRowSet was empty: no flush needed.
I20240501 08:12:31.287447 32640 tablet.cc:1722] T 12f63fbe69e144a8ae21d881c122ad7a P 91d11eae5ef6432ca86eb8ed446b6ea6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:31.287832 32640 tablet.cc:1620] T 12f63fbe69e144a8ae21d881c122ad7a P 91d11eae5ef6432ca86eb8ed446b6ea6: MemRowSet was empty: no flush needed.
I20240501 08:12:31.288882 32643 tablet.cc:1722] T 12f63fbe69e144a8ae21d881c122ad7a P aad81047009a4ffcbc450b60030583b5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:31.289304 32643 tablet.cc:1620] T 12f63fbe69e144a8ae21d881c122ad7a P aad81047009a4ffcbc450b60030583b5: MemRowSet was empty: no flush needed.
I20240501 08:12:31.298247 27618 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=5865465fbe8b4b5f8004e22f8f68f462] alter complete (version 2)
I20240501 08:12:31.405853 27616 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:31.415369 32638 tablet.cc:1722] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:31.416707 32643 tablet.cc:1722] T 12f63fbe69e144a8ae21d881c122ad7a P aad81047009a4ffcbc450b60030583b5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:31.417070 32640 tablet.cc:1722] T 12f63fbe69e144a8ae21d881c122ad7a P 91d11eae5ef6432ca86eb8ed446b6ea6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:31.429383 27613 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table_new_name [id=5865465fbe8b4b5f8004e22f8f68f462] alter complete (version 3)
I20240501 08:12:31.501053 32500 consensus_queue.cc:1035] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:31.519086 32444 consensus_queue.cc:1035] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:31.535836 27613 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:31.536187 27613 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:31.539652 27613 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 12f63fbe69e144a8ae21d881c122ad7a
I20240501 08:12:31.540246 28131 tablet_service.cc:1514] Processing DeleteTablet for tablet 12f63fbe69e144a8ae21d881c122ad7a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:31 UTC) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:31.540513 28002 tablet_service.cc:1514] Processing DeleteTablet for tablet 12f63fbe69e144a8ae21d881c122ad7a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:31 UTC) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:31.541139 32652 tablet_replica.cc:330] T 12f63fbe69e144a8ae21d881c122ad7a P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:31.541214 32651 tablet_replica.cc:330] T 12f63fbe69e144a8ae21d881c122ad7a P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:31.541460 32652 raft_consensus.cc:2229] T 12f63fbe69e144a8ae21d881c122ad7a P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:31.541744 27871 tablet_service.cc:1514] Processing DeleteTablet for tablet 12f63fbe69e144a8ae21d881c122ad7a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:31 UTC) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:31.541718 32651 raft_consensus.cc:2229] T 12f63fbe69e144a8ae21d881c122ad7a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:31.542048 32651 raft_consensus.cc:2258] T 12f63fbe69e144a8ae21d881c122ad7a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:31.543251 32651 ts_tablet_manager.cc:1875] T 12f63fbe69e144a8ae21d881c122ad7a P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:31.546154 32652 raft_consensus.cc:2258] T 12f63fbe69e144a8ae21d881c122ad7a P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:31.546723 32653 tablet_replica.cc:330] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:31.547061 32653 raft_consensus.cc:2229] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:31.547274 32652 ts_tablet_manager.cc:1875] T 12f63fbe69e144a8ae21d881c122ad7a P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:31.547498 32653 raft_consensus.cc:2258] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:31.559005 32653 ts_tablet_manager.cc:1875] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:31.561416 32652 ts_tablet_manager.cc:1888] T 12f63fbe69e144a8ae21d881c122ad7a P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:31.561705 32652 log.cc:1196] T 12f63fbe69e144a8ae21d881c122ad7a P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/12f63fbe69e144a8ae21d881c122ad7a
I20240501 08:12:31.562451 32652 ts_tablet_manager.cc:1909] T 12f63fbe69e144a8ae21d881c122ad7a P aad81047009a4ffcbc450b60030583b5: Deleting consensus metadata
I20240501 08:12:31.563742 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet 12f63fbe69e144a8ae21d881c122ad7a (table rebalancer_dynamic_table_new_name [id=5865465fbe8b4b5f8004e22f8f68f462]) successfully deleted
I20240501 08:12:31.565131 32653 ts_tablet_manager.cc:1888] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:31.565263 32653 log.cc:1196] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/12f63fbe69e144a8ae21d881c122ad7a
I20240501 08:12:31.565644 32653 ts_tablet_manager.cc:1909] T 12f63fbe69e144a8ae21d881c122ad7a P f1525cbc82a24d13b65f136695c903af: Deleting consensus metadata
I20240501 08:12:31.567041 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet 12f63fbe69e144a8ae21d881c122ad7a (table rebalancer_dynamic_table_new_name [id=5865465fbe8b4b5f8004e22f8f68f462]) successfully deleted
I20240501 08:12:31.579160 32651 ts_tablet_manager.cc:1888] T 12f63fbe69e144a8ae21d881c122ad7a P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:31.579339 32651 log.cc:1196] T 12f63fbe69e144a8ae21d881c122ad7a P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/12f63fbe69e144a8ae21d881c122ad7a
I20240501 08:12:31.579691 32651 ts_tablet_manager.cc:1909] T 12f63fbe69e144a8ae21d881c122ad7a P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting consensus metadata
I20240501 08:12:31.580966 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet 12f63fbe69e144a8ae21d881c122ad7a (table rebalancer_dynamic_table_new_name [id=5865465fbe8b4b5f8004e22f8f68f462]) successfully deleted
I20240501 08:12:31.599184 27613 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_extra_table_58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:31.602454 32564 consensus_queue.cc:1035] T b0906abad5c14912b94e9c017d2ecf8d P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:31.612466 27739 tablet_service.cc:1467] Processing CreateTablet for tablet 448a24d99bbe4fe4a571dc49a04ee275 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=7d3b03da535e4c14ad4122a4e27c19b5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:31.612968 27739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 448a24d99bbe4fe4a571dc49a04ee275. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:31.613121 28131 tablet_service.cc:1467] Processing CreateTablet for tablet b0eb3ba3e6e347ebaba87323f01cd8d4 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=7d3b03da535e4c14ad4122a4e27c19b5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:31.613462 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0eb3ba3e6e347ebaba87323f01cd8d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:31.613441 30915 tablet_service.cc:1467] Processing CreateTablet for tablet 448a24d99bbe4fe4a571dc49a04ee275 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=7d3b03da535e4c14ad4122a4e27c19b5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:31.613919 30915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 448a24d99bbe4fe4a571dc49a04ee275. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:31.612504 27741 tablet_service.cc:1467] Processing CreateTablet for tablet b0eb3ba3e6e347ebaba87323f01cd8d4 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=7d3b03da535e4c14ad4122a4e27c19b5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:31.617558 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0eb3ba3e6e347ebaba87323f01cd8d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:31.620455 28002 tablet_service.cc:1467] Processing CreateTablet for tablet 2ce43dc81a1b42c3a63124490d8e60cd (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=7d3b03da535e4c14ad4122a4e27c19b5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:31.620946 28002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ce43dc81a1b42c3a63124490d8e60cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:31.624040 30784 tablet_service.cc:1467] Processing CreateTablet for tablet b0eb3ba3e6e347ebaba87323f01cd8d4 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=7d3b03da535e4c14ad4122a4e27c19b5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:31.624603 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0eb3ba3e6e347ebaba87323f01cd8d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:31.624887 27871 tablet_service.cc:1467] Processing CreateTablet for tablet 2ce43dc81a1b42c3a63124490d8e60cd (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=7d3b03da535e4c14ad4122a4e27c19b5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:31.625373 27871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ce43dc81a1b42c3a63124490d8e60cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:31.626400 32509 consensus_queue.cc:1035] T b0906abad5c14912b94e9c017d2ecf8d P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:31.634428 30652 tablet_service.cc:1467] Processing CreateTablet for tablet 448a24d99bbe4fe4a571dc49a04ee275 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=7d3b03da535e4c14ad4122a4e27c19b5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:31.635007 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 448a24d99bbe4fe4a571dc49a04ee275. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:31.643301 27616 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:31.656687 30916 tablet_service.cc:1467] Processing CreateTablet for tablet 2ce43dc81a1b42c3a63124490d8e60cd (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=7d3b03da535e4c14ad4122a4e27c19b5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:31.657231 30916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ce43dc81a1b42c3a63124490d8e60cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:31.676266 28000 tablet_service.cc:1467] Processing CreateTablet for tablet 6d832814a0054420a086e4f0c3db3895 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5615d6f99f4d4590b7409ea0f881b3c8]), partition=
I20240501 08:12:31.676635 28000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d832814a0054420a086e4f0c3db3895. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:31.682933 30514 tablet_service.cc:1467] Processing CreateTablet for tablet 6d832814a0054420a086e4f0c3db3895 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5615d6f99f4d4590b7409ea0f881b3c8]), partition=
I20240501 08:12:31.683318 30514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d832814a0054420a086e4f0c3db3895. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:31.709535 32662 tablet_bootstrap.cc:492] T 448a24d99bbe4fe4a571dc49a04ee275 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:31.711323 32662 tablet_bootstrap.cc:654] T 448a24d99bbe4fe4a571dc49a04ee275 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:31.717571 32660 tablet_bootstrap.cc:492] T 6d832814a0054420a086e4f0c3db3895 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:31.719388 32660 tablet_bootstrap.cc:654] T 6d832814a0054420a086e4f0c3db3895 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:31.722926 32665 tablet_bootstrap.cc:492] T 6d832814a0054420a086e4f0c3db3895 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:31.723860 32661 tablet_bootstrap.cc:492] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:31.724792 32665 tablet_bootstrap.cc:654] T 6d832814a0054420a086e4f0c3db3895 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:31.726212 32661 tablet_bootstrap.cc:654] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:31.726426 32662 tablet_bootstrap.cc:492] T 448a24d99bbe4fe4a571dc49a04ee275 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:31.726606 32662 ts_tablet_manager.cc:1367] T 448a24d99bbe4fe4a571dc49a04ee275 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
W20240501 08:12:31.727331 27697 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.130:36503, user_credentials={real_user=slave}} blocked reactor thread for 68332.1us
I20240501 08:12:31.728338 32662 raft_consensus.cc:348] T 448a24d99bbe4fe4a571dc49a04ee275 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:31.729115 32663 tablet_bootstrap.cc:492] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:31.730005 32662 raft_consensus.cc:374] T 448a24d99bbe4fe4a571dc49a04ee275 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:31.730145 32662 raft_consensus.cc:729] T 448a24d99bbe4fe4a571dc49a04ee275 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:31.730401 32662 consensus_queue.cc:260] T 448a24d99bbe4fe4a571dc49a04ee275 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:31.731055 32662 ts_tablet_manager.cc:1398] T 448a24d99bbe4fe4a571dc49a04ee275 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 08:12:31.731081 32663 tablet_bootstrap.cc:654] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:31.732002 32662 tablet_bootstrap.cc:492] T 2ce43dc81a1b42c3a63124490d8e60cd P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:31.737246 32666 tablet_bootstrap.cc:492] T 2ce43dc81a1b42c3a63124490d8e60cd P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:31.738310 32664 tablet_bootstrap.cc:492] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:31.740208 32664 tablet_bootstrap.cc:654] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:31.741477 27741 tablet_service.cc:1467] Processing CreateTablet for tablet 6d832814a0054420a086e4f0c3db3895 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5615d6f99f4d4590b7409ea0f881b3c8]), partition=
I20240501 08:12:31.741904 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d832814a0054420a086e4f0c3db3895. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:31.742583 32662 tablet_bootstrap.cc:654] T 2ce43dc81a1b42c3a63124490d8e60cd P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:31.747921 32660 tablet_bootstrap.cc:492] T 6d832814a0054420a086e4f0c3db3895 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:31.748183 32660 ts_tablet_manager.cc:1367] T 6d832814a0054420a086e4f0c3db3895 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.002s
I20240501 08:12:31.749226 32660 raft_consensus.cc:348] T 6d832814a0054420a086e4f0c3db3895 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:31.749629 32660 raft_consensus.cc:374] T 6d832814a0054420a086e4f0c3db3895 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:31.749799 32660 raft_consensus.cc:729] T 6d832814a0054420a086e4f0c3db3895 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:31.750211 32660 consensus_queue.cc:260] T 6d832814a0054420a086e4f0c3db3895 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:31.752766 32660 ts_tablet_manager.cc:1398] T 6d832814a0054420a086e4f0c3db3895 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:31.754328 32664 tablet_bootstrap.cc:492] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:31.756489 32664 ts_tablet_manager.cc:1367] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.006s
I20240501 08:12:31.757426 32664 raft_consensus.cc:348] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:31.757707 32664 raft_consensus.cc:374] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:31.757839 32664 raft_consensus.cc:729] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:31.758194 32664 consensus_queue.cc:260] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:31.759254 32664 ts_tablet_manager.cc:1398] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:31.774524 32666 tablet_bootstrap.cc:654] T 2ce43dc81a1b42c3a63124490d8e60cd P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:31.777206 32661 tablet_bootstrap.cc:492] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:12:31.777534 32661 ts_tablet_manager.cc:1367] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.000s
I20240501 08:12:31.778389 32661 raft_consensus.cc:348] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:31.778671 32661 raft_consensus.cc:374] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:31.778736 32661 raft_consensus.cc:729] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:12:31.779021 32661 consensus_queue.cc:260] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:31.779527 32661 ts_tablet_manager.cc:1398] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:31.780659 32663 tablet_bootstrap.cc:492] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:31.781153 32663 ts_tablet_manager.cc:1367] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.052s user 0.000s sys 0.003s
I20240501 08:12:31.781708 32663 raft_consensus.cc:348] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:31.782104 32629 raft_consensus.cc:482] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:31.782266 32629 raft_consensus.cc:504] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:31.782984 32629 leader_election.cc:288] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:31.783964 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "448a24d99bbe4fe4a571dc49a04ee275" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:12:31.784695 30935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "448a24d99bbe4fe4a571dc49a04ee275" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:31.784865 30935 raft_consensus.cc:2454] T 448a24d99bbe4fe4a571dc49a04ee275 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 0.
W20240501 08:12:31.785332 30607 leader_election.cc:341] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:31.786793 30607 leader_election.cc:302] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a022e1a501f4d44a0468054d6c3de47, a29cd590999b4f32849ff5689cd53c11; no voters: 6331f77dae9d452184e8cd0cbf5f1b43
I20240501 08:12:31.787427 32629 raft_consensus.cc:2781] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:31.787534 32629 raft_consensus.cc:482] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:31.787614 32629 raft_consensus.cc:3037] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:31.788522 32665 tablet_bootstrap.cc:492] T 6d832814a0054420a086e4f0c3db3895 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:12:31.788658 32665 ts_tablet_manager.cc:1367] T 6d832814a0054420a086e4f0c3db3895 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.066s user 0.003s sys 0.009s
I20240501 08:12:31.791378 32665 raft_consensus.cc:348] T 6d832814a0054420a086e4f0c3db3895 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:31.791590 32665 raft_consensus.cc:374] T 6d832814a0054420a086e4f0c3db3895 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:31.791671 32665 raft_consensus.cc:729] T 6d832814a0054420a086e4f0c3db3895 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:12:31.791939 32665 consensus_queue.cc:260] T 6d832814a0054420a086e4f0c3db3895 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:31.792515 32665 ts_tablet_manager.cc:1398] T 6d832814a0054420a086e4f0c3db3895 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240501 08:12:31.793012 32665 tablet_bootstrap.cc:492] T 2ce43dc81a1b42c3a63124490d8e60cd P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:31.793982 32629 raft_consensus.cc:504] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:31.795070 32665 tablet_bootstrap.cc:654] T 2ce43dc81a1b42c3a63124490d8e60cd P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:31.794996 30935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "448a24d99bbe4fe4a571dc49a04ee275" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:31.795207 30935 raft_consensus.cc:3037] T 448a24d99bbe4fe4a571dc49a04ee275 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:31.808228 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "448a24d99bbe4fe4a571dc49a04ee275" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
W20240501 08:12:31.809115 30607 leader_election.cc:341] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:31.809522 32629 leader_election.cc:288] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 election: Requested vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:31.811978 30935 raft_consensus.cc:2454] T 448a24d99bbe4fe4a571dc49a04ee275 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 1.
I20240501 08:12:31.812656 30607 leader_election.cc:302] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a022e1a501f4d44a0468054d6c3de47, a29cd590999b4f32849ff5689cd53c11; no voters: 6331f77dae9d452184e8cd0cbf5f1b43
I20240501 08:12:31.813241 32629 raft_consensus.cc:2781] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:31.813905 32629 raft_consensus.cc:686] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Becoming Leader. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Running, Role: LEADER
I20240501 08:12:31.814368 32629 consensus_queue.cc:237] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:31.781893 32663 raft_consensus.cc:374] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:31.827698 32663 raft_consensus.cc:729] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:31.828161 32663 consensus_queue.cc:260] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:31.829335 32663 ts_tablet_manager.cc:1398] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.048s user 0.002s sys 0.000s
I20240501 08:12:31.829062 27613 catalog_manager.cc:5484] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134). New cstate: current_term: 1 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:31.830829 32666 tablet_bootstrap.cc:492] T 2ce43dc81a1b42c3a63124490d8e60cd P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:12:31.831037 32666 ts_tablet_manager.cc:1367] T 2ce43dc81a1b42c3a63124490d8e60cd P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.094s user 0.004s sys 0.000s
I20240501 08:12:31.832109 32666 raft_consensus.cc:348] T 2ce43dc81a1b42c3a63124490d8e60cd P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:31.832473 32666 raft_consensus.cc:374] T 2ce43dc81a1b42c3a63124490d8e60cd P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:31.832727 32666 raft_consensus.cc:729] T 2ce43dc81a1b42c3a63124490d8e60cd P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:12:31.833070 32666 consensus_queue.cc:260] T 2ce43dc81a1b42c3a63124490d8e60cd P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:31.833915 32666 ts_tablet_manager.cc:1398] T 2ce43dc81a1b42c3a63124490d8e60cd P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:31.834467 32655 tablet_bootstrap.cc:492] T 448a24d99bbe4fe4a571dc49a04ee275 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:31.837249 32662 tablet_bootstrap.cc:492] T 2ce43dc81a1b42c3a63124490d8e60cd P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:31.837417 32662 ts_tablet_manager.cc:1367] T 2ce43dc81a1b42c3a63124490d8e60cd P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.105s user 0.007s sys 0.023s
I20240501 08:12:31.837965 32655 tablet_bootstrap.cc:654] T 448a24d99bbe4fe4a571dc49a04ee275 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:31.838164 32662 raft_consensus.cc:348] T 2ce43dc81a1b42c3a63124490d8e60cd P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:31.838398 32662 raft_consensus.cc:374] T 2ce43dc81a1b42c3a63124490d8e60cd P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:31.838485 32662 raft_consensus.cc:729] T 2ce43dc81a1b42c3a63124490d8e60cd P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:31.838711 32662 consensus_queue.cc:260] T 2ce43dc81a1b42c3a63124490d8e60cd P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:31.839265 32662 ts_tablet_manager.cc:1398] T 2ce43dc81a1b42c3a63124490d8e60cd P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:31.840958 32665 tablet_bootstrap.cc:492] T 2ce43dc81a1b42c3a63124490d8e60cd P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:12:31.841176 32665 ts_tablet_manager.cc:1367] T 2ce43dc81a1b42c3a63124490d8e60cd P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.026s
I20240501 08:12:31.842129 32665 raft_consensus.cc:348] T 2ce43dc81a1b42c3a63124490d8e60cd P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:31.842965 32665 raft_consensus.cc:374] T 2ce43dc81a1b42c3a63124490d8e60cd P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:31.843052 32665 raft_consensus.cc:729] T 2ce43dc81a1b42c3a63124490d8e60cd P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:12:31.843287 32665 consensus_queue.cc:260] T 2ce43dc81a1b42c3a63124490d8e60cd P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:31.843853 32665 ts_tablet_manager.cc:1398] T 2ce43dc81a1b42c3a63124490d8e60cd P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:31.848435 32655 tablet_bootstrap.cc:492] T 448a24d99bbe4fe4a571dc49a04ee275 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:12:31.848587 32655 ts_tablet_manager.cc:1367] T 448a24d99bbe4fe4a571dc49a04ee275 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20240501 08:12:31.849398 32655 raft_consensus.cc:348] T 448a24d99bbe4fe4a571dc49a04ee275 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:31.849639 32655 raft_consensus.cc:374] T 448a24d99bbe4fe4a571dc49a04ee275 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:31.849704 32655 raft_consensus.cc:729] T 448a24d99bbe4fe4a571dc49a04ee275 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:12:31.849912 32655 consensus_queue.cc:260] T 448a24d99bbe4fe4a571dc49a04ee275 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } }
I20240501 08:12:31.851729 32655 ts_tablet_manager.cc:1398] T 448a24d99bbe4fe4a571dc49a04ee275 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:31.851981 32655 tablet_bootstrap.cc:492] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:31.853309 32655 tablet_bootstrap.cc:654] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:31.856165 32655 tablet_bootstrap.cc:492] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:12:31.856329 32655 ts_tablet_manager.cc:1367] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:31.856348 32657 raft_consensus.cc:482] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:31.856509 32657 raft_consensus.cc:504] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:31.856987 32655 raft_consensus.cc:348] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:31.857196 32655 raft_consensus.cc:374] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:31.857283 32657 leader_election.cc:288] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:31.857270 32655 raft_consensus.cc:729] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:12:31.857693 32655 consensus_queue.cc:260] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:31.857812 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0eb3ba3e6e347ebaba87323f01cd8d4" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:12:31.858024 27761 raft_consensus.cc:2454] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:12:31.858239 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0eb3ba3e6e347ebaba87323f01cd8d4" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:31.858425 28087 leader_election.cc:302] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:12:31.858546 30804 raft_consensus.cc:2454] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:12:31.858672 32657 raft_consensus.cc:2781] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:31.858739 32657 raft_consensus.cc:482] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:31.858795 32657 raft_consensus.cc:3037] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:31.859380 32655 ts_tablet_manager.cc:1398] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 08:12:31.859654 32655 tablet_bootstrap.cc:492] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:31.860886 32655 tablet_bootstrap.cc:654] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:31.861383 32657 raft_consensus.cc:504] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:31.861924 32657 leader_election.cc:288] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:31.862514 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0eb3ba3e6e347ebaba87323f01cd8d4" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:31.862607 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0eb3ba3e6e347ebaba87323f01cd8d4" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:12:31.862746 30804 raft_consensus.cc:3037] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:31.862773 27761 raft_consensus.cc:3037] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:31.865444 30804 raft_consensus.cc:2454] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:12:31.866708 27761 raft_consensus.cc:2454] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:12:31.867107 28086 leader_election.cc:302] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:12:31.868101 32658 raft_consensus.cc:2781] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:31.868256 32658 raft_consensus.cc:686] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:12:31.868563 32658 consensus_queue.cc:237] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:12:31.870101 32655 tablet_bootstrap.cc:492] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:12:31.870257 32655 ts_tablet_manager.cc:1367] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20240501 08:12:31.870911 32655 raft_consensus.cc:348] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:31.871104 32655 raft_consensus.cc:374] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:31.871179 32655 raft_consensus.cc:729] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:12:31.871361 32655 consensus_queue.cc:260] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:31.871897 32655 ts_tablet_manager.cc:1398] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:31.877156 27613 catalog_manager.cc:5484] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:32.042846 32079 raft_consensus.cc:482] T 2ce43dc81a1b42c3a63124490d8e60cd P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:32.043155 32079 raft_consensus.cc:504] T 2ce43dc81a1b42c3a63124490d8e60cd P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:32.044163 32079 leader_election.cc:288] T 2ce43dc81a1b42c3a63124490d8e60cd P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:32.044600 30935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ce43dc81a1b42c3a63124490d8e60cd" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:32.044701 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ce43dc81a1b42c3a63124490d8e60cd" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:12:32.044884 30935 raft_consensus.cc:2454] T 2ce43dc81a1b42c3a63124490d8e60cd P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:12:32.044906 28022 raft_consensus.cc:2454] T 2ce43dc81a1b42c3a63124490d8e60cd P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 0.
I20240501 08:12:32.045253 27827 leader_election.cc:302] T 2ce43dc81a1b42c3a63124490d8e60cd P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a29cd590999b4f32849ff5689cd53c11, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:32.045547 32079 raft_consensus.cc:2781] T 2ce43dc81a1b42c3a63124490d8e60cd P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:32.045637 32079 raft_consensus.cc:482] T 2ce43dc81a1b42c3a63124490d8e60cd P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:32.045734 32079 raft_consensus.cc:3037] T 2ce43dc81a1b42c3a63124490d8e60cd P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:32.048957 32079 raft_consensus.cc:504] T 2ce43dc81a1b42c3a63124490d8e60cd P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:32.049491 32079 leader_election.cc:288] T 2ce43dc81a1b42c3a63124490d8e60cd P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Requested vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:32.050097 30935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ce43dc81a1b42c3a63124490d8e60cd" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:32.050181 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ce43dc81a1b42c3a63124490d8e60cd" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:12:32.050330 30935 raft_consensus.cc:3037] T 2ce43dc81a1b42c3a63124490d8e60cd P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:32.050395 28022 raft_consensus.cc:3037] T 2ce43dc81a1b42c3a63124490d8e60cd P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:32.052593 28022 raft_consensus.cc:2454] T 2ce43dc81a1b42c3a63124490d8e60cd P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:12:32.052996 27827 leader_election.cc:302] T 2ce43dc81a1b42c3a63124490d8e60cd P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aad81047009a4ffcbc450b60030583b5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:32.053064 30935 raft_consensus.cc:2454] T 2ce43dc81a1b42c3a63124490d8e60cd P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 1.
I20240501 08:12:32.053294 32079 raft_consensus.cc:2781] T 2ce43dc81a1b42c3a63124490d8e60cd P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:32.053488 32079 raft_consensus.cc:686] T 2ce43dc81a1b42c3a63124490d8e60cd P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Becoming Leader. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Running, Role: LEADER
I20240501 08:12:32.053838 32079 consensus_queue.cc:237] T 2ce43dc81a1b42c3a63124490d8e60cd P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:32.058354 27613 catalog_manager.cc:5484] T 2ce43dc81a1b42c3a63124490d8e60cd P f1525cbc82a24d13b65f136695c903af reported cstate change: term changed from 0 to 1, leader changed from <none> to f1525cbc82a24d13b65f136695c903af (127.21.84.130). New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:32.063992 32509 raft_consensus.cc:482] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:32.064121 32509 raft_consensus.cc:504] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:32.064715 32509 leader_election.cc:288] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:32.065140 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d832814a0054420a086e4f0c3db3895" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:12:32.065274 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d832814a0054420a086e4f0c3db3895" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:32.065363 28022 raft_consensus.cc:2454] T 6d832814a0054420a086e4f0c3db3895 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:12:32.065522 30537 raft_consensus.cc:2454] T 6d832814a0054420a086e4f0c3db3895 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 0.
I20240501 08:12:32.065812 27696 leader_election.cc:302] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:12:32.066041 32509 raft_consensus.cc:2781] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:32.066119 32509 raft_consensus.cc:482] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:32.066183 32509 raft_consensus.cc:3037] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:32.068150 32509 raft_consensus.cc:504] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:32.069275 32509 leader_election.cc:288] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Requested vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:32.069808 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d832814a0054420a086e4f0c3db3895" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:32.069830 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d832814a0054420a086e4f0c3db3895" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:12:32.070071 28022 raft_consensus.cc:3037] T 6d832814a0054420a086e4f0c3db3895 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:32.070080 30537 raft_consensus.cc:3037] T 6d832814a0054420a086e4f0c3db3895 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:32.072775 30537 raft_consensus.cc:2454] T 6d832814a0054420a086e4f0c3db3895 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:12:32.072775 28022 raft_consensus.cc:2454] T 6d832814a0054420a086e4f0c3db3895 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 1.
I20240501 08:12:32.073215 27697 leader_election.cc:302] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 6331f77dae9d452184e8cd0cbf5f1b43; no voters:
I20240501 08:12:32.073489 32509 raft_consensus.cc:2781] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:32.073649 32509 raft_consensus.cc:686] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:12:32.073894 32509 consensus_queue.cc:237] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:32.077700 27613 catalog_manager.cc:5484] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:32.223274 27761 raft_consensus.cc:3037] T 448a24d99bbe4fe4a571dc49a04ee275 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:32.225749 32629 consensus_queue.cc:1035] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:32.232280 32580 consensus_queue.cc:1035] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:32.271522 27613 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:32.277447 30537 raft_consensus.cc:1261] T 6d832814a0054420a086e4f0c3db3895 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:32.278551 28021 raft_consensus.cc:1261] T 6d832814a0054420a086e4f0c3db3895 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:32.278519 32509 consensus_queue.cc:1035] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:32.279518 32509 consensus_queue.cc:1035] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:32.289521 32694 tablet.cc:1722] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:32.290149 32694 tablet.cc:1620] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43: MemRowSet was empty: no flush needed.
I20240501 08:12:32.291079 32695 tablet.cc:1722] T 6d832814a0054420a086e4f0c3db3895 P aad81047009a4ffcbc450b60030583b5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:32.291592 27613 catalog_manager.cc:5576] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305) does not have the latest schema for tablet 6d832814a0054420a086e4f0c3db3895 (table rebalancer_dynamic_table [id=5615d6f99f4d4590b7409ea0f881b3c8]). Expected version 1 got 0
I20240501 08:12:32.291970 32695 tablet.cc:1620] T 6d832814a0054420a086e4f0c3db3895 P aad81047009a4ffcbc450b60030583b5: MemRowSet was empty: no flush needed.
I20240501 08:12:32.298915 32658 consensus_queue.cc:1035] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:32.299726 27618 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=5615d6f99f4d4590b7409ea0f881b3c8] alter complete (version 1)
I20240501 08:12:32.310451 27618 catalog_manager.cc:5576] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293) does not have the latest schema for tablet 6d832814a0054420a086e4f0c3db3895 (table rebalancer_dynamic_table [id=5615d6f99f4d4590b7409ea0f881b3c8]). Expected version 1 got 0
I20240501 08:12:32.318368 32697 tablet.cc:1722] T 6d832814a0054420a086e4f0c3db3895 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:32.319904 32697 tablet.cc:1620] T 6d832814a0054420a086e4f0c3db3895 P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:32.324119 27613 catalog_manager.cc:5576] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293) does not have the latest schema for tablet 6d832814a0054420a086e4f0c3db3895 (table rebalancer_dynamic_table [id=5615d6f99f4d4590b7409ea0f881b3c8]). Expected version 1 got 0
I20240501 08:12:32.328248 32658 consensus_queue.cc:1035] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:32.403604 27613 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:32.413208 32695 tablet.cc:1722] T 6d832814a0054420a086e4f0c3db3895 P aad81047009a4ffcbc450b60030583b5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:32.413895 32695 tablet.cc:1620] T 6d832814a0054420a086e4f0c3db3895 P aad81047009a4ffcbc450b60030583b5: MemRowSet was empty: no flush needed.
I20240501 08:12:32.414278 32694 tablet.cc:1722] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:32.414681 32694 tablet.cc:1620] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43: MemRowSet was empty: no flush needed.
I20240501 08:12:32.415244 32697 tablet.cc:1722] T 6d832814a0054420a086e4f0c3db3895 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:32.415701 32697 tablet.cc:1620] T 6d832814a0054420a086e4f0c3db3895 P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:32.426452 27618 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=5615d6f99f4d4590b7409ea0f881b3c8] alter complete (version 2)
I20240501 08:12:32.509832 32079 consensus_queue.cc:1035] T 2ce43dc81a1b42c3a63124490d8e60cd P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:32.517431 32704 consensus_queue.cc:1035] T 2ce43dc81a1b42c3a63124490d8e60cd P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:32.529675 27613 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:32.537431 32694 tablet.cc:1722] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:32.539788 32695 tablet.cc:1722] T 6d832814a0054420a086e4f0c3db3895 P aad81047009a4ffcbc450b60030583b5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:32.541397 32697 tablet.cc:1722] T 6d832814a0054420a086e4f0c3db3895 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:32.552661 27613 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table_new_name [id=5615d6f99f4d4590b7409ea0f881b3c8] alter complete (version 3)
I20240501 08:12:32.614357 27618 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_extra_table_60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:32.623760 30652 tablet_service.cc:1467] Processing CreateTablet for tablet 00205a461b704a19bbd39bc71361e529 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=9a256e3f732a4230901aea19c32440ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:32.624186 30650 tablet_service.cc:1467] Processing CreateTablet for tablet 59ad966f760143cf9063e05a0079868c (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=9a256e3f732a4230901aea19c32440ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:32.624568 28131 tablet_service.cc:1467] Processing CreateTablet for tablet 59ad966f760143cf9063e05a0079868c (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=9a256e3f732a4230901aea19c32440ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:32.624970 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59ad966f760143cf9063e05a0079868c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:32.625391 30784 tablet_service.cc:1467] Processing CreateTablet for tablet 00205a461b704a19bbd39bc71361e529 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=9a256e3f732a4230901aea19c32440ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:32.625591 30514 tablet_service.cc:1467] Processing CreateTablet for tablet 59ad966f760143cf9063e05a0079868c (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=9a256e3f732a4230901aea19c32440ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:32.625730 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00205a461b704a19bbd39bc71361e529. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:32.626022 30514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59ad966f760143cf9063e05a0079868c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:32.628525 27871 tablet_service.cc:1467] Processing CreateTablet for tablet a3f0322a212a459b925a0f4e2dd2457e (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=9a256e3f732a4230901aea19c32440ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:32.628954 27871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3f0322a212a459b925a0f4e2dd2457e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:32.624325 30652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00205a461b704a19bbd39bc71361e529. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:32.631332 30650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59ad966f760143cf9063e05a0079868c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:32.634302 28000 tablet_service.cc:1467] Processing CreateTablet for tablet a3f0322a212a459b925a0f4e2dd2457e (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=9a256e3f732a4230901aea19c32440ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:32.634742 28000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3f0322a212a459b925a0f4e2dd2457e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:32.635357 30915 tablet_service.cc:1467] Processing CreateTablet for tablet 00205a461b704a19bbd39bc71361e529 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=9a256e3f732a4230901aea19c32440ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:32.635780 30915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00205a461b704a19bbd39bc71361e529. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:32.642395 30916 tablet_service.cc:1467] Processing CreateTablet for tablet a3f0322a212a459b925a0f4e2dd2457e (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=9a256e3f732a4230901aea19c32440ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:32.642868 30916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3f0322a212a459b925a0f4e2dd2457e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:32.645576 32708 tablet_bootstrap.cc:492] T a3f0322a212a459b925a0f4e2dd2457e P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:32.648489 32710 tablet_bootstrap.cc:492] T a3f0322a212a459b925a0f4e2dd2457e P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:32.651175 32708 tablet_bootstrap.cc:654] T a3f0322a212a459b925a0f4e2dd2457e P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:32.651732 32709 tablet_bootstrap.cc:492] T 59ad966f760143cf9063e05a0079868c P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:32.653604 32710 tablet_bootstrap.cc:654] T a3f0322a212a459b925a0f4e2dd2457e P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:32.653806 32709 tablet_bootstrap.cc:654] T 59ad966f760143cf9063e05a0079868c P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:32.655534 32708 tablet_bootstrap.cc:492] T a3f0322a212a459b925a0f4e2dd2457e P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:12:32.655750 32708 ts_tablet_manager.cc:1367] T a3f0322a212a459b925a0f4e2dd2457e P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240501 08:12:32.657243 32708 raft_consensus.cc:348] T a3f0322a212a459b925a0f4e2dd2457e P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:32.657591 32708 raft_consensus.cc:374] T a3f0322a212a459b925a0f4e2dd2457e P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:32.657474 27618 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:32.657743 32708 raft_consensus.cc:729] T a3f0322a212a459b925a0f4e2dd2457e P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:12:32.657747 27618 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:32.658113 32708 consensus_queue.cc:260] T a3f0322a212a459b925a0f4e2dd2457e P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:32.659253 32707 tablet_bootstrap.cc:492] T 59ad966f760143cf9063e05a0079868c P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:32.660308 32708 ts_tablet_manager.cc:1398] T a3f0322a212a459b925a0f4e2dd2457e P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 08:12:32.662653 27618 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 6d832814a0054420a086e4f0c3db3895
I20240501 08:12:32.664170 27739 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d832814a0054420a086e4f0c3db3895 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:32 UTC) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:32.665030 30514 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d832814a0054420a086e4f0c3db3895 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:32 UTC) from {username='slave'} at 127.0.0.1:45078
I20240501 08:12:32.665246 32707 tablet_bootstrap.cc:654] T 59ad966f760143cf9063e05a0079868c P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:32.667780 28000 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d832814a0054420a086e4f0c3db3895 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:32 UTC) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:32.668754 32710 tablet_bootstrap.cc:492] T a3f0322a212a459b925a0f4e2dd2457e P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:12:32.669456 32710 ts_tablet_manager.cc:1367] T a3f0322a212a459b925a0f4e2dd2457e P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20240501 08:12:32.671064 32710 raft_consensus.cc:348] T a3f0322a212a459b925a0f4e2dd2457e P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:32.672483 32710 raft_consensus.cc:374] T a3f0322a212a459b925a0f4e2dd2457e P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:32.672775 32710 raft_consensus.cc:729] T a3f0322a212a459b925a0f4e2dd2457e P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:12:32.673326 32710 consensus_queue.cc:260] T a3f0322a212a459b925a0f4e2dd2457e P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:32.676339 32710 ts_tablet_manager.cc:1398] T a3f0322a212a459b925a0f4e2dd2457e P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 08:12:32.679600 32707 tablet_bootstrap.cc:492] T 59ad966f760143cf9063e05a0079868c P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:32.679917 32707 ts_tablet_manager.cc:1367] T 59ad966f760143cf9063e05a0079868c P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.004s
I20240501 08:12:32.680773 32709 tablet_bootstrap.cc:492] T 59ad966f760143cf9063e05a0079868c P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:32.681690 32707 raft_consensus.cc:348] T 59ad966f760143cf9063e05a0079868c P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:32.682108 32709 ts_tablet_manager.cc:1367] T 59ad966f760143cf9063e05a0079868c P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.030s user 0.009s sys 0.000s
I20240501 08:12:32.682979 32709 raft_consensus.cc:348] T 59ad966f760143cf9063e05a0079868c P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:32.683231 32707 raft_consensus.cc:374] T 59ad966f760143cf9063e05a0079868c P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:32.683406 32707 raft_consensus.cc:729] T 59ad966f760143cf9063e05a0079868c P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:32.683547 32709 raft_consensus.cc:374] T 59ad966f760143cf9063e05a0079868c P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:32.683872 32707 consensus_queue.cc:260] T 59ad966f760143cf9063e05a0079868c P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:32.684207 32709 raft_consensus.cc:729] T 59ad966f760143cf9063e05a0079868c P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:32.684571 32709 consensus_queue.cc:260] T 59ad966f760143cf9063e05a0079868c P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:32.688017 32718 tablet_bootstrap.cc:492] T 59ad966f760143cf9063e05a0079868c P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:32.693017 32709 ts_tablet_manager.cc:1398] T 59ad966f760143cf9063e05a0079868c P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 08:12:32.693226 32707 ts_tablet_manager.cc:1398] T 59ad966f760143cf9063e05a0079868c P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20240501 08:12:32.699937 32707 tablet_bootstrap.cc:492] T 00205a461b704a19bbd39bc71361e529 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:32.704775 32717 tablet_replica.cc:330] T 6d832814a0054420a086e4f0c3db3895 P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:32.705252 32717 raft_consensus.cc:2229] T 6d832814a0054420a086e4f0c3db3895 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:32.706341 32717 raft_consensus.cc:2258] T 6d832814a0054420a086e4f0c3db3895 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:32.706604 32714 tablet_replica.cc:330] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:32.706997 32714 raft_consensus.cc:2229] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:32.708786 32707 tablet_bootstrap.cc:654] T 00205a461b704a19bbd39bc71361e529 P 9a022e1a501f4d44a0468054d6c3de47: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:32.710410 32714 raft_consensus.cc:2258] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:32.712563 32717 ts_tablet_manager.cc:1875] T 6d832814a0054420a086e4f0c3db3895 P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:32.714991 32714 ts_tablet_manager.cc:1875] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:32.715740 32719 tablet_bootstrap.cc:492] T a3f0322a212a459b925a0f4e2dd2457e P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:32.719286 32718 tablet_bootstrap.cc:654] T 59ad966f760143cf9063e05a0079868c P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:32.722790 32719 tablet_bootstrap.cc:654] T a3f0322a212a459b925a0f4e2dd2457e P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:32.728235 32715 tablet_replica.cc:330] T 6d832814a0054420a086e4f0c3db3895 P 31811708c6774037b82df44024b229b6: stopping tablet replica
I20240501 08:12:32.728925 32715 raft_consensus.cc:2229] T 6d832814a0054420a086e4f0c3db3895 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:32.729269 32715 raft_consensus.cc:2258] T 6d832814a0054420a086e4f0c3db3895 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:32.731096 32715 ts_tablet_manager.cc:1875] T 6d832814a0054420a086e4f0c3db3895 P 31811708c6774037b82df44024b229b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:32.731122 32706 tablet_bootstrap.cc:492] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:32.733445 32706 tablet_bootstrap.cc:654] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:32.735805 32718 tablet_bootstrap.cc:492] T 59ad966f760143cf9063e05a0079868c P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:12:32.735985 32718 ts_tablet_manager.cc:1367] T 59ad966f760143cf9063e05a0079868c P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.000s
I20240501 08:12:32.736832 32718 raft_consensus.cc:348] T 59ad966f760143cf9063e05a0079868c P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:32.737098 32718 raft_consensus.cc:374] T 59ad966f760143cf9063e05a0079868c P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:32.737175 32718 raft_consensus.cc:729] T 59ad966f760143cf9063e05a0079868c P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:12:32.737417 32718 consensus_queue.cc:260] T 59ad966f760143cf9063e05a0079868c P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:32.738147 32718 ts_tablet_manager.cc:1398] T 59ad966f760143cf9063e05a0079868c P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:12:32.738976 32715 ts_tablet_manager.cc:1888] T 6d832814a0054420a086e4f0c3db3895 P 31811708c6774037b82df44024b229b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:32.739766 32715 log.cc:1196] T 6d832814a0054420a086e4f0c3db3895 P 31811708c6774037b82df44024b229b6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/wals/6d832814a0054420a086e4f0c3db3895
I20240501 08:12:32.740509 32715 ts_tablet_manager.cc:1909] T 6d832814a0054420a086e4f0c3db3895 P 31811708c6774037b82df44024b229b6: Deleting consensus metadata
I20240501 08:12:32.743129 27607 catalog_manager.cc:4830] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): tablet 6d832814a0054420a086e4f0c3db3895 (table rebalancer_dynamic_table_new_name [id=5615d6f99f4d4590b7409ea0f881b3c8]) successfully deleted
I20240501 08:12:32.748126 32706 tablet_bootstrap.cc:492] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:32.748342 32706 ts_tablet_manager.cc:1367] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240501 08:12:32.749118 32706 raft_consensus.cc:348] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:32.749423 32706 raft_consensus.cc:374] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:32.749584 32706 raft_consensus.cc:729] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:32.749907 32706 consensus_queue.cc:260] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:32.750908 32706 ts_tablet_manager.cc:1398] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:32.752775 32707 tablet_bootstrap.cc:492] T 00205a461b704a19bbd39bc71361e529 P 9a022e1a501f4d44a0468054d6c3de47: No bootstrap required, opened a new log
I20240501 08:12:32.753031 32707 ts_tablet_manager.cc:1367] T 00205a461b704a19bbd39bc71361e529 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.053s user 0.005s sys 0.017s
I20240501 08:12:32.753880 32707 raft_consensus.cc:348] T 00205a461b704a19bbd39bc71361e529 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:32.754215 32707 raft_consensus.cc:374] T 00205a461b704a19bbd39bc71361e529 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:32.754333 32707 raft_consensus.cc:729] T 00205a461b704a19bbd39bc71361e529 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: FOLLOWER
I20240501 08:12:32.754652 32707 consensus_queue.cc:260] T 00205a461b704a19bbd39bc71361e529 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:32.760686 32717 ts_tablet_manager.cc:1888] T 6d832814a0054420a086e4f0c3db3895 P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:32.761186 32293 raft_consensus.cc:482] T 59ad966f760143cf9063e05a0079868c P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:32.761335 32293 raft_consensus.cc:504] T 59ad966f760143cf9063e05a0079868c P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:32.762060 32293 leader_election.cc:288] T 59ad966f760143cf9063e05a0079868c P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:32.762822 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59ad966f760143cf9063e05a0079868c" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:32.763062 30537 raft_consensus.cc:2454] T 59ad966f760143cf9063e05a0079868c P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:12:32.763442 28088 leader_election.cc:302] T 59ad966f760143cf9063e05a0079868c P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:12:32.763695 32293 raft_consensus.cc:2781] T 59ad966f760143cf9063e05a0079868c P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:32.763761 32293 raft_consensus.cc:482] T 59ad966f760143cf9063e05a0079868c P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:32.763816 32293 raft_consensus.cc:3037] T 59ad966f760143cf9063e05a0079868c P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:32.766268 27618 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:32.767546 30671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59ad966f760143cf9063e05a0079868c" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:32.767756 30671 raft_consensus.cc:2454] T 59ad966f760143cf9063e05a0079868c P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:12:32.774930 32707 ts_tablet_manager.cc:1398] T 00205a461b704a19bbd39bc71361e529 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240501 08:12:32.781927 30514 tablet_service.cc:1467] Processing CreateTablet for tablet 4e840ca720594127b84c502d85a1f8d4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d8f726189a2147b4874bb3e31b67128c]), partition=
I20240501 08:12:32.781917 27871 tablet_service.cc:1467] Processing CreateTablet for tablet 4e840ca720594127b84c502d85a1f8d4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d8f726189a2147b4874bb3e31b67128c]), partition=
I20240501 08:12:32.782337 30514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e840ca720594127b84c502d85a1f8d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:32.782356 27871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e840ca720594127b84c502d85a1f8d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:32.782280 32293 raft_consensus.cc:504] T 59ad966f760143cf9063e05a0079868c P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:32.783876 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59ad966f760143cf9063e05a0079868c" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:32.783916 30671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59ad966f760143cf9063e05a0079868c" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:32.784109 30671 raft_consensus.cc:3037] T 59ad966f760143cf9063e05a0079868c P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:32.784149 30537 raft_consensus.cc:3037] T 59ad966f760143cf9063e05a0079868c P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:32.787416 30671 raft_consensus.cc:2454] T 59ad966f760143cf9063e05a0079868c P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:12:32.787855 28085 leader_election.cc:302] T 59ad966f760143cf9063e05a0079868c P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:32.788343 32657 raft_consensus.cc:2781] T 59ad966f760143cf9063e05a0079868c P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:32.788470 32657 raft_consensus.cc:686] T 59ad966f760143cf9063e05a0079868c P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:12:32.789019 32709 tablet_bootstrap.cc:492] T 4e840ca720594127b84c502d85a1f8d4 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:32.789642 30537 raft_consensus.cc:2454] T 59ad966f760143cf9063e05a0079868c P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:12:32.789902 28131 tablet_service.cc:1467] Processing CreateTablet for tablet 4e840ca720594127b84c502d85a1f8d4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d8f726189a2147b4874bb3e31b67128c]), partition=
I20240501 08:12:32.790169 32293 leader_election.cc:288] T 59ad966f760143cf9063e05a0079868c P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:32.790270 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e840ca720594127b84c502d85a1f8d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:32.788715 32657 consensus_queue.cc:237] T 59ad966f760143cf9063e05a0079868c P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:32.791154 32717 log.cc:1196] T 6d832814a0054420a086e4f0c3db3895 P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/6d832814a0054420a086e4f0c3db3895
I20240501 08:12:32.792447 32717 ts_tablet_manager.cc:1909] T 6d832814a0054420a086e4f0c3db3895 P aad81047009a4ffcbc450b60030583b5: Deleting consensus metadata
I20240501 08:12:32.793692 32709 tablet_bootstrap.cc:654] T 4e840ca720594127b84c502d85a1f8d4 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:32.793826 32714 ts_tablet_manager.cc:1888] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:32.793944 32714 log.cc:1196] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/6d832814a0054420a086e4f0c3db3895
I20240501 08:12:32.794401 32714 ts_tablet_manager.cc:1909] T 6d832814a0054420a086e4f0c3db3895 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting consensus metadata
I20240501 08:12:32.799021 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet 6d832814a0054420a086e4f0c3db3895 (table rebalancer_dynamic_table_new_name [id=5615d6f99f4d4590b7409ea0f881b3c8]) successfully deleted
I20240501 08:12:32.803651 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet 6d832814a0054420a086e4f0c3db3895 (table rebalancer_dynamic_table_new_name [id=5615d6f99f4d4590b7409ea0f881b3c8]) successfully deleted
I20240501 08:12:32.808493 32718 tablet_bootstrap.cc:492] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:32.810000 32718 tablet_bootstrap.cc:654] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:32.811504 32718 tablet_bootstrap.cc:492] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:12:32.811702 32718 ts_tablet_manager.cc:1367] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240501 08:12:32.812438 32718 raft_consensus.cc:348] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:32.812700 32718 raft_consensus.cc:374] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:32.812793 32718 raft_consensus.cc:729] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:12:32.813037 32718 consensus_queue.cc:260] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:32.813736 32718 ts_tablet_manager.cc:1398] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:32.816285 32708 tablet_bootstrap.cc:492] T 4e840ca720594127b84c502d85a1f8d4 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:32.816998 32719 tablet_bootstrap.cc:492] T a3f0322a212a459b925a0f4e2dd2457e P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:32.817173 32719 ts_tablet_manager.cc:1367] T a3f0322a212a459b925a0f4e2dd2457e P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.102s user 0.000s sys 0.017s
I20240501 08:12:32.817570 32708 tablet_bootstrap.cc:654] T 4e840ca720594127b84c502d85a1f8d4 P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:32.818034 32719 raft_consensus.cc:348] T a3f0322a212a459b925a0f4e2dd2457e P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:32.818250 32719 raft_consensus.cc:374] T a3f0322a212a459b925a0f4e2dd2457e P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:32.818339 32719 raft_consensus.cc:729] T a3f0322a212a459b925a0f4e2dd2457e P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:32.818540 32719 consensus_queue.cc:260] T a3f0322a212a459b925a0f4e2dd2457e P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:32.822005 32719 ts_tablet_manager.cc:1398] T a3f0322a212a459b925a0f4e2dd2457e P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 08:12:32.822345 32719 tablet_bootstrap.cc:492] T 00205a461b704a19bbd39bc71361e529 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:32.823843 32719 tablet_bootstrap.cc:654] T 00205a461b704a19bbd39bc71361e529 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:32.825716 32709 tablet_bootstrap.cc:492] T 4e840ca720594127b84c502d85a1f8d4 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:32.826122 32709 ts_tablet_manager.cc:1367] T 4e840ca720594127b84c502d85a1f8d4 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.037s user 0.013s sys 0.000s
I20240501 08:12:32.827302 32709 raft_consensus.cc:348] T 4e840ca720594127b84c502d85a1f8d4 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:32.827868 32709 raft_consensus.cc:374] T 4e840ca720594127b84c502d85a1f8d4 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:32.827984 32709 raft_consensus.cc:729] T 4e840ca720594127b84c502d85a1f8d4 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:32.828346 32709 consensus_queue.cc:260] T 4e840ca720594127b84c502d85a1f8d4 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:32.837105 32709 ts_tablet_manager.cc:1398] T 4e840ca720594127b84c502d85a1f8d4 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20240501 08:12:32.840358 32634 raft_consensus.cc:482] T a3f0322a212a459b925a0f4e2dd2457e P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:32.840579 32634 raft_consensus.cc:504] T a3f0322a212a459b925a0f4e2dd2457e P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:32.841607 32634 leader_election.cc:288] T a3f0322a212a459b925a0f4e2dd2457e P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:32.843144 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3f0322a212a459b925a0f4e2dd2457e" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:12:32.843354 32708 tablet_bootstrap.cc:492] T 4e840ca720594127b84c502d85a1f8d4 P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:12:32.843191 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3f0322a212a459b925a0f4e2dd2457e" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:12:32.843405 28021 raft_consensus.cc:2454] T a3f0322a212a459b925a0f4e2dd2457e P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:32.843564 27889 raft_consensus.cc:2454] T a3f0322a212a459b925a0f4e2dd2457e P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:32.843880 32708 ts_tablet_manager.cc:1367] T 4e840ca720594127b84c502d85a1f8d4 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.027s user 0.011s sys 0.002s
I20240501 08:12:32.843864 30872 leader_election.cc:302] T a3f0322a212a459b925a0f4e2dd2457e P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a29cd590999b4f32849ff5689cd53c11, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:12:32.844516 32309 raft_consensus.cc:2781] T a3f0322a212a459b925a0f4e2dd2457e P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:32.844638 32309 raft_consensus.cc:482] T a3f0322a212a459b925a0f4e2dd2457e P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:32.844728 32309 raft_consensus.cc:3037] T a3f0322a212a459b925a0f4e2dd2457e P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:32.845057 32708 raft_consensus.cc:348] T 4e840ca720594127b84c502d85a1f8d4 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:32.845273 32708 raft_consensus.cc:374] T 4e840ca720594127b84c502d85a1f8d4 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:32.845337 32708 raft_consensus.cc:729] T 4e840ca720594127b84c502d85a1f8d4 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:12:32.845571 32708 consensus_queue.cc:260] T 4e840ca720594127b84c502d85a1f8d4 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:32.847867 32708 ts_tablet_manager.cc:1398] T 4e840ca720594127b84c502d85a1f8d4 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 08:12:32.848691 32309 raft_consensus.cc:504] T a3f0322a212a459b925a0f4e2dd2457e P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:32.849375 32309 leader_election.cc:288] T a3f0322a212a459b925a0f4e2dd2457e P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Requested vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:32.849736 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3f0322a212a459b925a0f4e2dd2457e" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:12:32.849972 27889 raft_consensus.cc:3037] T a3f0322a212a459b925a0f4e2dd2457e P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:32.850160 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3f0322a212a459b925a0f4e2dd2457e" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:12:32.850383 28021 raft_consensus.cc:3037] T a3f0322a212a459b925a0f4e2dd2457e P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:32.853010 28021 raft_consensus.cc:2454] T a3f0322a212a459b925a0f4e2dd2457e P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:32.853731 27889 raft_consensus.cc:2454] T a3f0322a212a459b925a0f4e2dd2457e P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:32.854269 30873 leader_election.cc:302] T a3f0322a212a459b925a0f4e2dd2457e P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a29cd590999b4f32849ff5689cd53c11, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:32.862188 32309 raft_consensus.cc:2781] T a3f0322a212a459b925a0f4e2dd2457e P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:32.862392 32309 raft_consensus.cc:686] T a3f0322a212a459b925a0f4e2dd2457e P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Becoming Leader. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:12:32.862661 32309 consensus_queue.cc:237] T a3f0322a212a459b925a0f4e2dd2457e P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:32.864769 27618 catalog_manager.cc:5484] T 59ad966f760143cf9063e05a0079868c P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:32.869457 27620 catalog_manager.cc:5484] T a3f0322a212a459b925a0f4e2dd2457e P a29cd590999b4f32849ff5689cd53c11 reported cstate change: term changed from 0 to 1, leader changed from <none> to a29cd590999b4f32849ff5689cd53c11 (127.21.84.136). New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:32.874823 32719 tablet_bootstrap.cc:492] T 00205a461b704a19bbd39bc71361e529 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:32.875012 32719 ts_tablet_manager.cc:1367] T 00205a461b704a19bbd39bc71361e529 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.053s user 0.008s sys 0.031s
I20240501 08:12:32.875813 32719 raft_consensus.cc:348] T 00205a461b704a19bbd39bc71361e529 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:32.876050 32719 raft_consensus.cc:374] T 00205a461b704a19bbd39bc71361e529 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:32.876161 32719 raft_consensus.cc:729] T 00205a461b704a19bbd39bc71361e529 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:32.876386 32719 consensus_queue.cc:260] T 00205a461b704a19bbd39bc71361e529 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:32.876971 32719 ts_tablet_manager.cc:1398] T 00205a461b704a19bbd39bc71361e529 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:32.887658 32302 raft_consensus.cc:482] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:32.887849 32302 raft_consensus.cc:504] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:32.888484 32302 leader_election.cc:288] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:32.889081 30670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00205a461b704a19bbd39bc71361e529" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:12:32.889259 30670 raft_consensus.cc:2454] T 00205a461b704a19bbd39bc71361e529 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:32.889279 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00205a461b704a19bbd39bc71361e529" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:32.889533 30936 raft_consensus.cc:2454] T 00205a461b704a19bbd39bc71361e529 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 0.
I20240501 08:12:32.889623 30737 leader_election.cc:302] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:32.889974 32378 raft_consensus.cc:2781] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:32.890097 32378 raft_consensus.cc:482] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:32.890179 32378 raft_consensus.cc:3037] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:32.893751 32378 raft_consensus.cc:504] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:32.894338 32378 leader_election.cc:288] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Requested vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:32.895017 30670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00205a461b704a19bbd39bc71361e529" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:32.895045 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00205a461b704a19bbd39bc71361e529" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:32.895263 30670 raft_consensus.cc:3037] T 00205a461b704a19bbd39bc71361e529 P 9a022e1a501f4d44a0468054d6c3de47 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:32.895256 30936 raft_consensus.cc:3037] T 00205a461b704a19bbd39bc71361e529 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:32.897369 30670 raft_consensus.cc:2454] T 00205a461b704a19bbd39bc71361e529 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:32.898365 30737 leader_election.cc:302] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:12:32.898877 32378 raft_consensus.cc:2781] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:32.898001 30936 raft_consensus.cc:2454] T 00205a461b704a19bbd39bc71361e529 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:12:32.900630 32378 raft_consensus.cc:686] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Becoming Leader. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Running, Role: LEADER
I20240501 08:12:32.901075 32378 consensus_queue.cc:237] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:12:32.905604 27618 catalog_manager.cc:5484] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64998323a1f4482d8704bf20594d13a5 (127.21.84.135). New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:33.196341 32657 raft_consensus.cc:482] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:33.196537 32657 raft_consensus.cc:504] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:33.197140 32657 leader_election.cc:288] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:33.197544 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e840ca720594127b84c502d85a1f8d4" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:12:33.197780 27890 raft_consensus.cc:2454] T 4e840ca720594127b84c502d85a1f8d4 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:12:33.197729 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e840ca720594127b84c502d85a1f8d4" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:33.198004 30536 raft_consensus.cc:2454] T 4e840ca720594127b84c502d85a1f8d4 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:12:33.198176 28088 leader_election.cc:302] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:33.198405 32657 raft_consensus.cc:2781] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:33.198518 32657 raft_consensus.cc:482] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:33.198607 32657 raft_consensus.cc:3037] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:33.200932 32657 raft_consensus.cc:504] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:33.201402 32657 leader_election.cc:288] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:33.201826 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e840ca720594127b84c502d85a1f8d4" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:12:33.201817 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e840ca720594127b84c502d85a1f8d4" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:33.202057 27890 raft_consensus.cc:3037] T 4e840ca720594127b84c502d85a1f8d4 P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:33.202080 30536 raft_consensus.cc:3037] T 4e840ca720594127b84c502d85a1f8d4 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:33.204892 27890 raft_consensus.cc:2454] T 4e840ca720594127b84c502d85a1f8d4 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:12:33.205627 30536 raft_consensus.cc:2454] T 4e840ca720594127b84c502d85a1f8d4 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:12:33.205950 28088 leader_election.cc:302] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:33.206216 32657 raft_consensus.cc:2781] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:33.206336 32657 raft_consensus.cc:686] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:12:33.206589 32657 consensus_queue.cc:237] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:12:33.211841 27618 catalog_manager.cc:5484] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:33.218142 32657 consensus_queue.cc:1035] T 59ad966f760143cf9063e05a0079868c P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:33.225456 32657 consensus_queue.cc:1035] T 59ad966f760143cf9063e05a0079868c P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:33.330777 32731 consensus_queue.cc:1035] T a3f0322a212a459b925a0f4e2dd2457e P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:33.337996 32309 consensus_queue.cc:1035] T a3f0322a212a459b925a0f4e2dd2457e P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 08:12:33.340893 27616 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:33.348268 30537 raft_consensus.cc:1261] T 4e840ca720594127b84c502d85a1f8d4 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:33.348829 27890 raft_consensus.cc:1261] T 4e840ca720594127b84c502d85a1f8d4 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:33.349409 32293 consensus_queue.cc:1035] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:33.352022 32293 consensus_queue.cc:1035] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:33.374197 27618 catalog_manager.cc:5576] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503) does not have the latest schema for tablet 4e840ca720594127b84c502d85a1f8d4 (table rebalancer_dynamic_table [id=d8f726189a2147b4874bb3e31b67128c]). Expected version 1 got 0
I20240501 08:12:33.374907 27616 catalog_manager.cc:5576] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293) does not have the latest schema for tablet 4e840ca720594127b84c502d85a1f8d4 (table rebalancer_dynamic_table [id=d8f726189a2147b4874bb3e31b67128c]). Expected version 1 got 0
I20240501 08:12:33.379557 32302 consensus_queue.cc:1035] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:33.398746 32749 tablet.cc:1722] T 4e840ca720594127b84c502d85a1f8d4 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:33.403048 32749 tablet.cc:1620] T 4e840ca720594127b84c502d85a1f8d4 P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:33.405108 27613 catalog_manager.cc:5576] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293) does not have the latest schema for tablet 4e840ca720594127b84c502d85a1f8d4 (table rebalancer_dynamic_table [id=d8f726189a2147b4874bb3e31b67128c]). Expected version 1 got 0
I20240501 08:12:33.410943 32747 tablet.cc:1722] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:33.411331 32748 tablet.cc:1722] T 4e840ca720594127b84c502d85a1f8d4 P f1525cbc82a24d13b65f136695c903af: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:33.412009 32748 tablet.cc:1620] T 4e840ca720594127b84c502d85a1f8d4 P f1525cbc82a24d13b65f136695c903af: MemRowSet was empty: no flush needed.
I20240501 08:12:33.412065 32747 tablet.cc:1620] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6: MemRowSet was empty: no flush needed.
I20240501 08:12:33.414340 27616 catalog_manager.cc:5576] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323) does not have the latest schema for tablet 4e840ca720594127b84c502d85a1f8d4 (table rebalancer_dynamic_table [id=d8f726189a2147b4874bb3e31b67128c]). Expected version 1 got 0
I20240501 08:12:33.414932 27613 catalog_manager.cc:5576] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503) does not have the latest schema for tablet 4e840ca720594127b84c502d85a1f8d4 (table rebalancer_dynamic_table [id=d8f726189a2147b4874bb3e31b67128c]). Expected version 1 got 0
I20240501 08:12:33.419831 27618 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39210:
name: "rebalancer_extra_table_62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:33.426713 27613 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=d8f726189a2147b4874bb3e31b67128c] alter complete (version 1)
I20240501 08:12:33.432248 30784 tablet_service.cc:1467] Processing CreateTablet for tablet 6137a463366349398eef28755d247e95 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=0f8f4d8350da4a678e64ef5f70c8df92]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:33.432866 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6137a463366349398eef28755d247e95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:33.447654 32378 consensus_queue.cc:1035] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:33.449190 28131 tablet_service.cc:1467] Processing CreateTablet for tablet 6137a463366349398eef28755d247e95 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=0f8f4d8350da4a678e64ef5f70c8df92]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:33.449700 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6137a463366349398eef28755d247e95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:33.465325 27739 tablet_service.cc:1467] Processing CreateTablet for tablet 946add184f4a46f28fd631e8a3bc0ba7 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=0f8f4d8350da4a678e64ef5f70c8df92]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:33.465425 30514 tablet_service.cc:1467] Processing CreateTablet for tablet 6137a463366349398eef28755d247e95 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=0f8f4d8350da4a678e64ef5f70c8df92]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:33.465818 27739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 946add184f4a46f28fd631e8a3bc0ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:33.465888 30514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6137a463366349398eef28755d247e95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:33.466013 30916 tablet_service.cc:1467] Processing CreateTablet for tablet 946add184f4a46f28fd631e8a3bc0ba7 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=0f8f4d8350da4a678e64ef5f70c8df92]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:33.466444 30916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 946add184f4a46f28fd631e8a3bc0ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:33.466862 27741 tablet_service.cc:1467] Processing CreateTablet for tablet 287ebcf5c8af4469af28e69b6008af0a (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=0f8f4d8350da4a678e64ef5f70c8df92]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:33.467192 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 287ebcf5c8af4469af28e69b6008af0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:33.470459 28000 tablet_service.cc:1467] Processing CreateTablet for tablet 946add184f4a46f28fd631e8a3bc0ba7 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=0f8f4d8350da4a678e64ef5f70c8df92]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:33.470887 28000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 946add184f4a46f28fd631e8a3bc0ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:33.471117 28002 tablet_service.cc:1467] Processing CreateTablet for tablet 287ebcf5c8af4469af28e69b6008af0a (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=0f8f4d8350da4a678e64ef5f70c8df92]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:33.471462 28002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 287ebcf5c8af4469af28e69b6008af0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:33.475112 27871 tablet_service.cc:1467] Processing CreateTablet for tablet 287ebcf5c8af4469af28e69b6008af0a (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=0f8f4d8350da4a678e64ef5f70c8df92]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:33.475539 27871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 287ebcf5c8af4469af28e69b6008af0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:33.504308 32757 tablet_bootstrap.cc:492] T 287ebcf5c8af4469af28e69b6008af0a P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:33.506644 32759 tablet_bootstrap.cc:492] T 6137a463366349398eef28755d247e95 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:33.509811 32762 tablet_bootstrap.cc:492] T 287ebcf5c8af4469af28e69b6008af0a P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:33.510527 32759 tablet_bootstrap.cc:654] T 6137a463366349398eef28755d247e95 P 91d11eae5ef6432ca86eb8ed446b6ea6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:33.512138 32760 tablet_bootstrap.cc:492] T 6137a463366349398eef28755d247e95 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:33.515762 32761 tablet_bootstrap.cc:492] T 946add184f4a46f28fd631e8a3bc0ba7 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:33.517789 32761 tablet_bootstrap.cc:654] T 946add184f4a46f28fd631e8a3bc0ba7 P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:33.517937 32762 tablet_bootstrap.cc:654] T 287ebcf5c8af4469af28e69b6008af0a P f1525cbc82a24d13b65f136695c903af: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:33.518774 32760 tablet_bootstrap.cc:654] T 6137a463366349398eef28755d247e95 P 64998323a1f4482d8704bf20594d13a5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:33.520790 32757 tablet_bootstrap.cc:654] T 287ebcf5c8af4469af28e69b6008af0a P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:33.521986 32758 tablet_bootstrap.cc:492] T 6137a463366349398eef28755d247e95 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:33.523576 32761 tablet_bootstrap.cc:492] T 946add184f4a46f28fd631e8a3bc0ba7 P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:12:33.526212 32756 tablet_bootstrap.cc:492] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:33.526762 32761 ts_tablet_manager.cc:1367] T 946add184f4a46f28fd631e8a3bc0ba7 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240501 08:12:33.532951 32761 raft_consensus.cc:348] T 946add184f4a46f28fd631e8a3bc0ba7 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:33.534456 32761 raft_consensus.cc:374] T 946add184f4a46f28fd631e8a3bc0ba7 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:33.534742 32761 raft_consensus.cc:729] T 946add184f4a46f28fd631e8a3bc0ba7 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:12:33.535200 32761 consensus_queue.cc:260] T 946add184f4a46f28fd631e8a3bc0ba7 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:33.535696 32756 tablet_bootstrap.cc:654] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:33.537817 32761 ts_tablet_manager.cc:1398] T 946add184f4a46f28fd631e8a3bc0ba7 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:12:33.538623 32761 tablet_bootstrap.cc:492] T 287ebcf5c8af4469af28e69b6008af0a P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:33.540596 32761 tablet_bootstrap.cc:654] T 287ebcf5c8af4469af28e69b6008af0a P aad81047009a4ffcbc450b60030583b5: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:33.540827 27617 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:33.563766 32536 raft_consensus.cc:482] T 946add184f4a46f28fd631e8a3bc0ba7 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:33.563938 32536 raft_consensus.cc:504] T 946add184f4a46f28fd631e8a3bc0ba7 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:33.563961 32759 tablet_bootstrap.cc:492] T 6137a463366349398eef28755d247e95 P 91d11eae5ef6432ca86eb8ed446b6ea6: No bootstrap required, opened a new log
I20240501 08:12:33.564771 32759 ts_tablet_manager.cc:1367] T 6137a463366349398eef28755d247e95 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.058s user 0.004s sys 0.000s
I20240501 08:12:33.564796 32536 leader_election.cc:288] T 946add184f4a46f28fd631e8a3bc0ba7 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:33.565224 30935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "946add184f4a46f28fd631e8a3bc0ba7" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:12:33.565371 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "946add184f4a46f28fd631e8a3bc0ba7" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
W20240501 08:12:33.565915 27957 leader_election.cc:341] T 946add184f4a46f28fd631e8a3bc0ba7 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): Illegal state: must be running to vote when last-logged opid is not known
W20240501 08:12:33.566207 27957 leader_election.cc:341] T 946add184f4a46f28fd631e8a3bc0ba7 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:33.566293 27957 leader_election.cc:302] T 946add184f4a46f28fd631e8a3bc0ba7 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aad81047009a4ffcbc450b60030583b5; no voters: 6331f77dae9d452184e8cd0cbf5f1b43, a29cd590999b4f32849ff5689cd53c11
I20240501 08:12:33.567484 32747 tablet.cc:1722] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:33.567970 32747 tablet.cc:1620] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6: MemRowSet was empty: no flush needed.
I20240501 08:12:33.568584 32758 tablet_bootstrap.cc:654] T 6137a463366349398eef28755d247e95 P 31811708c6774037b82df44024b229b6: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:33.571471 32757 tablet_bootstrap.cc:492] T 287ebcf5c8af4469af28e69b6008af0a P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:12:33.571682 32757 ts_tablet_manager.cc:1367] T 287ebcf5c8af4469af28e69b6008af0a P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.067s user 0.000s sys 0.004s
I20240501 08:12:33.572602 32761 tablet_bootstrap.cc:492] T 287ebcf5c8af4469af28e69b6008af0a P aad81047009a4ffcbc450b60030583b5: No bootstrap required, opened a new log
I20240501 08:12:33.573161 32761 ts_tablet_manager.cc:1367] T 287ebcf5c8af4469af28e69b6008af0a P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.000s
I20240501 08:12:33.574015 32761 raft_consensus.cc:348] T 287ebcf5c8af4469af28e69b6008af0a P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:33.574087 32757 raft_consensus.cc:348] T 287ebcf5c8af4469af28e69b6008af0a P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:33.574275 32757 raft_consensus.cc:374] T 287ebcf5c8af4469af28e69b6008af0a P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:33.574412 32757 raft_consensus.cc:729] T 287ebcf5c8af4469af28e69b6008af0a P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:12:33.574774 32757 consensus_queue.cc:260] T 287ebcf5c8af4469af28e69b6008af0a P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:33.575502 32761 raft_consensus.cc:374] T 287ebcf5c8af4469af28e69b6008af0a P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:33.575786 32757 ts_tablet_manager.cc:1398] T 287ebcf5c8af4469af28e69b6008af0a P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:33.575763 32761 raft_consensus.cc:729] T 287ebcf5c8af4469af28e69b6008af0a P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: FOLLOWER
I20240501 08:12:33.576103 32757 tablet_bootstrap.cc:492] T 946add184f4a46f28fd631e8a3bc0ba7 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:33.576041 32749 tablet.cc:1722] T 4e840ca720594127b84c502d85a1f8d4 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:33.576404 32536 raft_consensus.cc:2726] T 946add184f4a46f28fd631e8a3bc0ba7 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:33.576565 32749 tablet.cc:1620] T 4e840ca720594127b84c502d85a1f8d4 P 31811708c6774037b82df44024b229b6: MemRowSet was empty: no flush needed.
I20240501 08:12:33.577000 32748 tablet.cc:1722] T 4e840ca720594127b84c502d85a1f8d4 P f1525cbc82a24d13b65f136695c903af: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:33.577054 32761 consensus_queue.cc:260] T 287ebcf5c8af4469af28e69b6008af0a P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:33.577440 32748 tablet.cc:1620] T 4e840ca720594127b84c502d85a1f8d4 P f1525cbc82a24d13b65f136695c903af: MemRowSet was empty: no flush needed.
I20240501 08:12:33.579948 32761 ts_tablet_manager.cc:1398] T 287ebcf5c8af4469af28e69b6008af0a P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 08:12:33.581874 32760 tablet_bootstrap.cc:492] T 6137a463366349398eef28755d247e95 P 64998323a1f4482d8704bf20594d13a5: No bootstrap required, opened a new log
I20240501 08:12:33.582741 32760 ts_tablet_manager.cc:1367] T 6137a463366349398eef28755d247e95 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.070s user 0.004s sys 0.001s
I20240501 08:12:33.584905 32760 raft_consensus.cc:348] T 6137a463366349398eef28755d247e95 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:33.585223 32760 raft_consensus.cc:374] T 6137a463366349398eef28755d247e95 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:33.585363 32760 raft_consensus.cc:729] T 6137a463366349398eef28755d247e95 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: FOLLOWER
I20240501 08:12:33.585678 32760 consensus_queue.cc:260] T 6137a463366349398eef28755d247e95 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:33.586483 32760 ts_tablet_manager.cc:1398] T 6137a463366349398eef28755d247e95 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 08:12:33.587687 32759 raft_consensus.cc:348] T 6137a463366349398eef28755d247e95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:33.587917 32759 raft_consensus.cc:374] T 6137a463366349398eef28755d247e95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:33.587996 32759 raft_consensus.cc:729] T 6137a463366349398eef28755d247e95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: FOLLOWER
I20240501 08:12:33.588241 32759 consensus_queue.cc:260] T 6137a463366349398eef28755d247e95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:33.588948 32759 ts_tablet_manager.cc:1398] T 6137a463366349398eef28755d247e95 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20240501 08:12:33.589499 27616 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table [id=d8f726189a2147b4874bb3e31b67128c] alter complete (version 2)
I20240501 08:12:33.597429 32758 tablet_bootstrap.cc:492] T 6137a463366349398eef28755d247e95 P 31811708c6774037b82df44024b229b6: No bootstrap required, opened a new log
I20240501 08:12:33.597681 32758 ts_tablet_manager.cc:1367] T 6137a463366349398eef28755d247e95 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.076s user 0.000s sys 0.004s
I20240501 08:12:33.598496 32758 raft_consensus.cc:348] T 6137a463366349398eef28755d247e95 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:33.600373 32756 tablet_bootstrap.cc:492] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11: No bootstrap required, opened a new log
I20240501 08:12:33.600572 32756 ts_tablet_manager.cc:1367] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.074s user 0.004s sys 0.000s
I20240501 08:12:33.601347 32756 raft_consensus.cc:348] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:33.601570 32756 raft_consensus.cc:374] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:33.601691 32756 raft_consensus.cc:729] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: FOLLOWER
I20240501 08:12:33.601752 32762 tablet_bootstrap.cc:492] T 287ebcf5c8af4469af28e69b6008af0a P f1525cbc82a24d13b65f136695c903af: No bootstrap required, opened a new log
I20240501 08:12:33.601922 32762 ts_tablet_manager.cc:1367] T 287ebcf5c8af4469af28e69b6008af0a P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.092s user 0.003s sys 0.000s
I20240501 08:12:33.601987 32756 consensus_queue.cc:260] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:33.602541 32756 ts_tablet_manager.cc:1398] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:12:33.602737 32762 raft_consensus.cc:348] T 287ebcf5c8af4469af28e69b6008af0a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:33.602931 32762 raft_consensus.cc:374] T 287ebcf5c8af4469af28e69b6008af0a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:33.603001 32762 raft_consensus.cc:729] T 287ebcf5c8af4469af28e69b6008af0a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: FOLLOWER
I20240501 08:12:33.603200 32762 consensus_queue.cc:260] T 287ebcf5c8af4469af28e69b6008af0a P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:33.608666 32762 ts_tablet_manager.cc:1398] T 287ebcf5c8af4469af28e69b6008af0a P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 08:12:33.610455 32758 raft_consensus.cc:374] T 6137a463366349398eef28755d247e95 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:33.610626 32758 raft_consensus.cc:729] T 6137a463366349398eef28755d247e95 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: FOLLOWER
I20240501 08:12:33.610894 32309 raft_consensus.cc:482] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:33.611023 32309 raft_consensus.cc:504] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:33.611616 32309 leader_election.cc:288] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:33.612255 32757 tablet_bootstrap.cc:654] T 946add184f4a46f28fd631e8a3bc0ba7 P 6331f77dae9d452184e8cd0cbf5f1b43: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:33.612286 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "946add184f4a46f28fd631e8a3bc0ba7" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
W20240501 08:12:33.612771 30872 leader_election.cc:341] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:33.613263 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "946add184f4a46f28fd631e8a3bc0ba7" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:12:33.613236 32758 consensus_queue.cc:260] T 6137a463366349398eef28755d247e95 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:33.613472 28022 raft_consensus.cc:2454] T 946add184f4a46f28fd631e8a3bc0ba7 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 0.
I20240501 08:12:33.613797 32758 ts_tablet_manager.cc:1398] T 6137a463366349398eef28755d247e95 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.016s user 0.000s sys 0.001s
I20240501 08:12:33.615137 30872 leader_election.cc:302] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a29cd590999b4f32849ff5689cd53c11, aad81047009a4ffcbc450b60030583b5; no voters: 6331f77dae9d452184e8cd0cbf5f1b43
I20240501 08:12:33.615424 32309 raft_consensus.cc:2781] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:33.615568 32309 raft_consensus.cc:482] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:33.615654 32309 raft_consensus.cc:3037] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:33.615819 32757 tablet_bootstrap.cc:492] T 946add184f4a46f28fd631e8a3bc0ba7 P 6331f77dae9d452184e8cd0cbf5f1b43: No bootstrap required, opened a new log
I20240501 08:12:33.615971 32757 ts_tablet_manager.cc:1367] T 946add184f4a46f28fd631e8a3bc0ba7 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.001s
I20240501 08:12:33.616611 32757 raft_consensus.cc:348] T 946add184f4a46f28fd631e8a3bc0ba7 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:33.616847 32757 raft_consensus.cc:374] T 946add184f4a46f28fd631e8a3bc0ba7 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:33.616938 32757 raft_consensus.cc:729] T 946add184f4a46f28fd631e8a3bc0ba7 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: FOLLOWER
I20240501 08:12:33.617272 32757 consensus_queue.cc:260] T 946add184f4a46f28fd631e8a3bc0ba7 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:33.618391 32309 raft_consensus.cc:504] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:33.619002 32757 ts_tablet_manager.cc:1398] T 946add184f4a46f28fd631e8a3bc0ba7 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:12:33.619064 32309 leader_election.cc:288] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Requested vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:33.620774 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "946add184f4a46f28fd631e8a3bc0ba7" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:12:33.620903 27761 raft_consensus.cc:3037] T 946add184f4a46f28fd631e8a3bc0ba7 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:33.622360 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "946add184f4a46f28fd631e8a3bc0ba7" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:12:33.622566 28022 raft_consensus.cc:3037] T 946add184f4a46f28fd631e8a3bc0ba7 P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:33.622951 27761 raft_consensus.cc:2454] T 946add184f4a46f28fd631e8a3bc0ba7 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:33.623401 30872 leader_election.cc:302] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:33.625434 32309 raft_consensus.cc:2781] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:33.625624 28022 raft_consensus.cc:2454] T 946add184f4a46f28fd631e8a3bc0ba7 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:12:33.626183 32309 raft_consensus.cc:686] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Becoming Leader. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:12:33.626458 32309 consensus_queue.cc:237] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:12:33.630918 27619 catalog_manager.cc:5484] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: term changed from 0 to 1, leader changed from <none> to a29cd590999b4f32849ff5689cd53c11 (127.21.84.136). New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:33.670109 32536 raft_consensus.cc:482] T 287ebcf5c8af4469af28e69b6008af0a P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:33.670279 32536 raft_consensus.cc:504] T 287ebcf5c8af4469af28e69b6008af0a P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:33.670876 32536 leader_election.cc:288] T 287ebcf5c8af4469af28e69b6008af0a P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:12:33.671211 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "287ebcf5c8af4469af28e69b6008af0a" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" is_pre_election: true
I20240501 08:12:33.671311 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "287ebcf5c8af4469af28e69b6008af0a" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af" is_pre_election: true
I20240501 08:12:33.671396 27761 raft_consensus.cc:2454] T 287ebcf5c8af4469af28e69b6008af0a P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 0.
I20240501 08:12:33.671510 27890 raft_consensus.cc:2454] T 287ebcf5c8af4469af28e69b6008af0a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 0.
I20240501 08:12:33.671777 27957 leader_election.cc:302] T 287ebcf5c8af4469af28e69b6008af0a P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:12:33.672097 32536 raft_consensus.cc:2781] T 287ebcf5c8af4469af28e69b6008af0a P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:33.672215 32536 raft_consensus.cc:482] T 287ebcf5c8af4469af28e69b6008af0a P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:33.672284 32536 raft_consensus.cc:3037] T 287ebcf5c8af4469af28e69b6008af0a P aad81047009a4ffcbc450b60030583b5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:33.674222 32536 raft_consensus.cc:504] T 287ebcf5c8af4469af28e69b6008af0a P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:33.674705 32536 leader_election.cc:288] T 287ebcf5c8af4469af28e69b6008af0a P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Requested vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:12:33.675110 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "287ebcf5c8af4469af28e69b6008af0a" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:12:33.675107 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "287ebcf5c8af4469af28e69b6008af0a" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:12:33.675244 27890 raft_consensus.cc:3037] T 287ebcf5c8af4469af28e69b6008af0a P f1525cbc82a24d13b65f136695c903af [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:33.675263 27761 raft_consensus.cc:3037] T 287ebcf5c8af4469af28e69b6008af0a P 6331f77dae9d452184e8cd0cbf5f1b43 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:33.677045 27890 raft_consensus.cc:2454] T 287ebcf5c8af4469af28e69b6008af0a P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 1.
I20240501 08:12:33.677220 27761 raft_consensus.cc:2454] T 287ebcf5c8af4469af28e69b6008af0a P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 1.
I20240501 08:12:33.677356 27958 leader_election.cc:302] T 287ebcf5c8af4469af28e69b6008af0a P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aad81047009a4ffcbc450b60030583b5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:33.677858 32536 raft_consensus.cc:2781] T 287ebcf5c8af4469af28e69b6008af0a P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:33.678015 32536 raft_consensus.cc:686] T 287ebcf5c8af4469af28e69b6008af0a P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Becoming Leader. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Running, Role: LEADER
I20240501 08:12:33.678562 32536 consensus_queue.cc:237] T 287ebcf5c8af4469af28e69b6008af0a P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:12:33.682070 27619 catalog_manager.cc:5484] T 287ebcf5c8af4469af28e69b6008af0a P aad81047009a4ffcbc450b60030583b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to aad81047009a4ffcbc450b60030583b5 (127.21.84.131). New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:33.699712 27619 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:33.721976 32747 tablet.cc:1722] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:33.726090 32749 tablet.cc:1722] T 4e840ca720594127b84c502d85a1f8d4 P 31811708c6774037b82df44024b229b6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:33.731405 27619 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: rebalancer_dynamic_table_new_name [id=d8f726189a2147b4874bb3e31b67128c] alter complete (version 3)
I20240501 08:12:33.740928 32748 tablet.cc:1722] T 4e840ca720594127b84c502d85a1f8d4 P f1525cbc82a24d13b65f136695c903af: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:33.770174 32658 raft_consensus.cc:482] T 6137a463366349398eef28755d247e95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:33.770339 32658 raft_consensus.cc:504] T 6137a463366349398eef28755d247e95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:33.770922 32658 leader_election.cc:288] T 6137a463366349398eef28755d247e95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:33.771526 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6137a463366349398eef28755d247e95" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:12:33.771555 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6137a463366349398eef28755d247e95" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:12:33.771751 30804 raft_consensus.cc:2454] T 6137a463366349398eef28755d247e95 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:12:33.771777 30537 raft_consensus.cc:2454] T 6137a463366349398eef28755d247e95 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 0.
I20240501 08:12:33.772234 28086 leader_election.cc:302] T 6137a463366349398eef28755d247e95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:12:33.773028 32658 raft_consensus.cc:2781] T 6137a463366349398eef28755d247e95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:33.773353 32658 raft_consensus.cc:482] T 6137a463366349398eef28755d247e95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:33.773428 32658 raft_consensus.cc:3037] T 6137a463366349398eef28755d247e95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:33.776008 32658 raft_consensus.cc:504] T 6137a463366349398eef28755d247e95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:33.780704 32658 leader_election.cc:288] T 6137a463366349398eef28755d247e95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Requested vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:33.781178 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6137a463366349398eef28755d247e95" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:33.781360 30804 raft_consensus.cc:3037] T 6137a463366349398eef28755d247e95 P 64998323a1f4482d8704bf20594d13a5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:33.782035 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6137a463366349398eef28755d247e95" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:33.782277 30537 raft_consensus.cc:3037] T 6137a463366349398eef28755d247e95 P 31811708c6774037b82df44024b229b6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:33.783717 30804 raft_consensus.cc:2454] T 6137a463366349398eef28755d247e95 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:12:33.785233 30537 raft_consensus.cc:2454] T 6137a463366349398eef28755d247e95 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:12:33.785662 28088 leader_election.cc:302] T 6137a463366349398eef28755d247e95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:12:33.786005 32658 raft_consensus.cc:2781] T 6137a463366349398eef28755d247e95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:33.786103 32658 raft_consensus.cc:686] T 6137a463366349398eef28755d247e95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:12:33.786408 32658 consensus_queue.cc:237] T 6137a463366349398eef28755d247e95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } }
I20240501 08:12:33.797454 27619 catalog_manager.cc:5484] T 6137a463366349398eef28755d247e95 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:33.833050 27619 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:33.833261 27619 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39210:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:33.836820 27619 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 30c0f17c356e49fca6e933d80d164d83: Sending DeleteTablet for 3 replicas of tablet 4e840ca720594127b84c502d85a1f8d4
I20240501 08:12:33.837654 27871 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e840ca720594127b84c502d85a1f8d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:33 UTC) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:33.837764 30514 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e840ca720594127b84c502d85a1f8d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:33 UTC) from {username='slave'} at 127.0.0.1:45078
I20240501 08:12:33.838338 28131 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e840ca720594127b84c502d85a1f8d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:33 UTC) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:33.844822 310 tablet_replica.cc:330] T 4e840ca720594127b84c502d85a1f8d4 P 31811708c6774037b82df44024b229b6: stopping tablet replica
I20240501 08:12:33.845662 310 raft_consensus.cc:2229] T 4e840ca720594127b84c502d85a1f8d4 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:33.846282 310 raft_consensus.cc:2258] T 4e840ca720594127b84c502d85a1f8d4 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:33.846460 311 tablet_replica.cc:330] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:33.846686 311 raft_consensus.cc:2229] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:33.847678 310 ts_tablet_manager.cc:1875] T 4e840ca720594127b84c502d85a1f8d4 P 31811708c6774037b82df44024b229b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:33.849149 311 raft_consensus.cc:2258] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:33.851133 311 ts_tablet_manager.cc:1875] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:33.853031 309 tablet_replica.cc:330] T 4e840ca720594127b84c502d85a1f8d4 P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:33.853446 309 raft_consensus.cc:2229] T 4e840ca720594127b84c502d85a1f8d4 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:33.853762 309 raft_consensus.cc:2258] T 4e840ca720594127b84c502d85a1f8d4 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:33.855134 310 ts_tablet_manager.cc:1888] T 4e840ca720594127b84c502d85a1f8d4 P 31811708c6774037b82df44024b229b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:33.855906 309 ts_tablet_manager.cc:1875] T 4e840ca720594127b84c502d85a1f8d4 P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:33.856918 310 log.cc:1196] T 4e840ca720594127b84c502d85a1f8d4 P 31811708c6774037b82df44024b229b6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/wals/4e840ca720594127b84c502d85a1f8d4
I20240501 08:12:33.858366 311 ts_tablet_manager.cc:1888] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:33.858521 311 log.cc:1196] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/4e840ca720594127b84c502d85a1f8d4
I20240501 08:12:33.858486 310 ts_tablet_manager.cc:1909] T 4e840ca720594127b84c502d85a1f8d4 P 31811708c6774037b82df44024b229b6: Deleting consensus metadata
I20240501 08:12:33.858997 311 ts_tablet_manager.cc:1909] T 4e840ca720594127b84c502d85a1f8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting consensus metadata
I20240501 08:12:33.860291 27607 catalog_manager.cc:4830] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): tablet 4e840ca720594127b84c502d85a1f8d4 (table rebalancer_dynamic_table_new_name [id=d8f726189a2147b4874bb3e31b67128c]) successfully deleted
I20240501 08:12:33.860284 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet 4e840ca720594127b84c502d85a1f8d4 (table rebalancer_dynamic_table_new_name [id=d8f726189a2147b4874bb3e31b67128c]) successfully deleted
I20240501 08:12:33.864668 309 ts_tablet_manager.cc:1888] T 4e840ca720594127b84c502d85a1f8d4 P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:33.864851 309 log.cc:1196] T 4e840ca720594127b84c502d85a1f8d4 P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/4e840ca720594127b84c502d85a1f8d4
I20240501 08:12:33.865442 309 ts_tablet_manager.cc:1909] T 4e840ca720594127b84c502d85a1f8d4 P f1525cbc82a24d13b65f136695c903af: Deleting consensus metadata
I20240501 08:12:33.866520 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet 4e840ca720594127b84c502d85a1f8d4 (table rebalancer_dynamic_table_new_name [id=d8f726189a2147b4874bb3e31b67128c]) successfully deleted
I20240501 08:12:34.143049 32634 consensus_queue.cc:1035] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:34.152331 32309 consensus_queue.cc:1035] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:34.210117 32536 consensus_queue.cc:1035] T 287ebcf5c8af4469af28e69b6008af0a P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:34.216341 32536 consensus_queue.cc:1035] T 287ebcf5c8af4469af28e69b6008af0a P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:34.239419 307 consensus_queue.cc:1035] T 6137a463366349398eef28755d247e95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:34.259737 32293 consensus_queue.cc:1035] T 6137a463366349398eef28755d247e95 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:34.817567 27741 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:12:34.824697 30784 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 08:12:34.860471 28131 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 08:12:34.877413 30650 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:34.895962 30514 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:12:34.903581 28000 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:12:34.966781 30916 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:12:34.991356 27871 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:12:35.432454 27741 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:12:35.463896 28131 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 08:12:35.468428 30784 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 08:12:35.473731 30916 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:12:35.518805 30514 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:12:35.540611 30650 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:35.580127 28000 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:12:35.611454 27871 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:12:36.136435 30784 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 08:12:36.143208 28131 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 08:12:36.164518 30650 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:36.168625 27741 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:12:36.170210 30514 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:12:36.211954 28000 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:12:36.229661 27871 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:12:36.236547 30916 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:12:36.540020 30803 consensus_queue.cc:237] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:12:36.543329 30669 raft_consensus.cc:1261] T 00a73d23b12d48a2966f94789db4369a P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:36.543689 30936 raft_consensus.cc:1261] T 00a73d23b12d48a2966f94789db4369a P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:12:36.543952 30737 consensus_peers.cc:485] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 -> Peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Couldn't send request to peer 91d11eae5ef6432ca86eb8ed446b6ea6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00a73d23b12d48a2966f94789db4369a. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:36.544376 32302 consensus_queue.cc:1035] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:36.544970 32378 consensus_queue.cc:1035] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:36.550856 32378 raft_consensus.cc:2932] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:12:36.557461 30669 raft_consensus.cc:2932] T 00a73d23b12d48a2966f94789db4369a P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:12:36.559234 30936 raft_consensus.cc:2932] T 00a73d23b12d48a2966f94789db4369a P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:12:36.565371 27615 catalog_manager.cc:5484] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:36.599807 27761 consensus_queue.cc:237] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:12:36.602840 27889 raft_consensus.cc:1261] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:36.602782 28021 raft_consensus.cc:1261] T 802538381de843ab8d326014a0b4ec15 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:36.603479 312 consensus_queue.cc:1035] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:12:36.604682 27694 consensus_peers.cc:485] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 -> Peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Couldn't send request to peer 9a022e1a501f4d44a0468054d6c3de47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 802538381de843ab8d326014a0b4ec15. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:36.605160 32701 consensus_queue.cc:1035] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:36.611093 312 raft_consensus.cc:2932] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:36.614938 28021 raft_consensus.cc:2932] T 802538381de843ab8d326014a0b4ec15 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:36.617668 27889 raft_consensus.cc:2932] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:36.626642 27612 catalog_manager.cc:5484] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index -1 to 4, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:36.642438 30669 consensus_queue.cc:237] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:12:36.644968 30936 raft_consensus.cc:1261] T 683aa9595f2e456c943fc4d16f22e83c P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:36.645052 30803 raft_consensus.cc:1261] T 683aa9595f2e456c943fc4d16f22e83c P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:36.645694 516 consensus_queue.cc:1035] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:36.646860 362 consensus_queue.cc:1035] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:12:36.647146 30607 consensus_peers.cc:485] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 683aa9595f2e456c943fc4d16f22e83c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:36.650231 362 raft_consensus.cc:2932] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:36.651571 30936 raft_consensus.cc:2932] T 683aa9595f2e456c943fc4d16f22e83c P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:36.651818 30803 raft_consensus.cc:2932] T 683aa9595f2e456c943fc4d16f22e83c P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:36.657783 27615 catalog_manager.cc:5484] T 683aa9595f2e456c943fc4d16f22e83c P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New cstate: current_term: 1 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:36.682420 30803 consensus_queue.cc:237] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:36.685142 30669 raft_consensus.cc:1261] T 131eac1e19264e00b76500d2ba42697a P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:36.685168 30537 raft_consensus.cc:1261] T 131eac1e19264e00b76500d2ba42697a P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:12:36.685633 30740 consensus_peers.cc:485] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 131eac1e19264e00b76500d2ba42697a. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:36.685917 32302 consensus_queue.cc:1035] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:36.686486 550 consensus_queue.cc:1035] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:36.689464 32302 raft_consensus.cc:2932] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:36.690490 30671 raft_consensus.cc:2932] T 131eac1e19264e00b76500d2ba42697a P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:36.690599 30537 raft_consensus.cc:2932] T 131eac1e19264e00b76500d2ba42697a P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:36.699836 27619 catalog_manager.cc:5484] T 131eac1e19264e00b76500d2ba42697a P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:36.750157 27889 consensus_queue.cc:237] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:12:36.753481 30936 raft_consensus.cc:1261] T 4fd595c79cdc43f9a8afadb6b548bf64 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:12:36.753814 27825 consensus_peers.cc:485] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af -> Peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Couldn't send request to peer 91d11eae5ef6432ca86eb8ed446b6ea6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4fd595c79cdc43f9a8afadb6b548bf64. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:36.754065 455 consensus_queue.cc:1035] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:36.755014 30671 raft_consensus.cc:1261] T 4fd595c79cdc43f9a8afadb6b548bf64 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:36.755595 455 consensus_queue.cc:1035] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:36.776254 455 raft_consensus.cc:2932] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:12:36.776780 30671 raft_consensus.cc:2932] T 4fd595c79cdc43f9a8afadb6b548bf64 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:12:36.783658 27612 catalog_manager.cc:5484] T 4fd595c79cdc43f9a8afadb6b548bf64 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:12:36.800189 30936 raft_consensus.cc:2932] T 4fd595c79cdc43f9a8afadb6b548bf64 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:12:36.815956 30936 consensus_queue.cc:237] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
W20240501 08:12:36.819746 30870 consensus_peers.cc:485] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 -> Peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Couldn't send request to peer 9a022e1a501f4d44a0468054d6c3de47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 946add184f4a46f28fd631e8a3bc0ba7. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:36.819890 28021 raft_consensus.cc:1261] T 946add184f4a46f28fd631e8a3bc0ba7 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:36.819993 27761 raft_consensus.cc:1261] T 946add184f4a46f28fd631e8a3bc0ba7 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:36.820490 585 consensus_queue.cc:1035] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:36.820881 32634 consensus_queue.cc:1035] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:36.823779 456 raft_consensus.cc:2932] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:36.826748 27761 raft_consensus.cc:2932] T 946add184f4a46f28fd631e8a3bc0ba7 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:36.829984 27612 catalog_manager.cc:5484] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index -1 to 2, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:36.830601 28021 raft_consensus.cc:2932] T 946add184f4a46f28fd631e8a3bc0ba7 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:36.850755 27759 consensus_queue.cc:237] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
I20240501 08:12:36.853109 28152 raft_consensus.cc:1261] T 8ab0410a98974002ab68dd0f6a456d59 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:36.853380 30936 raft_consensus.cc:1261] T 8ab0410a98974002ab68dd0f6a456d59 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:12:36.853576 27695 consensus_peers.cc:485] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 -> Peer 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): Couldn't send request to peer 64998323a1f4482d8704bf20594d13a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ab0410a98974002ab68dd0f6a456d59. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:36.853642 563 consensus_queue.cc:1035] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:36.853883 32701 consensus_queue.cc:1035] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:36.856813 312 raft_consensus.cc:2932] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:12:36.857518 30936 raft_consensus.cc:2932] T 8ab0410a98974002ab68dd0f6a456d59 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:12:36.863261 28152 raft_consensus.cc:2932] T 8ab0410a98974002ab68dd0f6a456d59 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:12:36.865516 27612 catalog_manager.cc:5484] T 8ab0410a98974002ab68dd0f6a456d59 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index -1 to 2, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:12:36.887953 27759 consensus_queue.cc:237] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:36.890776 30537 raft_consensus.cc:1261] T 103e1394bfe34a86ba1d9487d75fd512 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:36.890857 30671 raft_consensus.cc:1261] T 103e1394bfe34a86ba1d9487d75fd512 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:12:36.890911 27697 consensus_peers.cc:485] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 103e1394bfe34a86ba1d9487d75fd512. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:36.891405 312 consensus_queue.cc:1035] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:36.891884 312 consensus_queue.cc:1035] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:36.895848 312 raft_consensus.cc:2932] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:36.896661 30537 raft_consensus.cc:2932] T 103e1394bfe34a86ba1d9487d75fd512 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:36.897112 30671 raft_consensus.cc:2932] T 103e1394bfe34a86ba1d9487d75fd512 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:36.903894 27619 catalog_manager.cc:5484] T 103e1394bfe34a86ba1d9487d75fd512 P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:36.954691 27889 consensus_queue.cc:237] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:12:36.957304 30936 raft_consensus.cc:1261] T c4851258508d46c88493503a6257f365 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:36.957479 30803 raft_consensus.cc:1261] T c4851258508d46c88493503a6257f365 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:12:36.957736 27825 consensus_peers.cc:485] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af -> Peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Couldn't send request to peer 91d11eae5ef6432ca86eb8ed446b6ea6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c4851258508d46c88493503a6257f365. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:36.958459 32079 consensus_queue.cc:1035] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:36.958705 596 consensus_queue.cc:1035] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:36.963850 596 raft_consensus.cc:2932] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:12:36.969170 30803 raft_consensus.cc:2932] T c4851258508d46c88493503a6257f365 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:12:36.969300 30936 raft_consensus.cc:2932] T c4851258508d46c88493503a6257f365 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:12:36.970587 27615 catalog_manager.cc:5484] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af reported cstate change: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:36.990628 28152 consensus_queue.cc:237] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:12:36.993466 28021 raft_consensus.cc:1261] T 5536362915034f3eb86138b45d71b8ad P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:36.993507 27889 raft_consensus.cc:1261] T 5536362915034f3eb86138b45d71b8ad P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:12:36.993811 28085 consensus_peers.cc:485] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6 -> Peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Couldn't send request to peer 9a022e1a501f4d44a0468054d6c3de47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5536362915034f3eb86138b45d71b8ad. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:36.994246 307 consensus_queue.cc:1035] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:36.994765 584 consensus_queue.cc:1035] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:36.997659 307 raft_consensus.cc:2932] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:36.998260 27889 raft_consensus.cc:2932] T 5536362915034f3eb86138b45d71b8ad P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:37.001271 28021 raft_consensus.cc:2932] T 5536362915034f3eb86138b45d71b8ad P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:37.006799 27615 catalog_manager.cc:5484] T 5536362915034f3eb86138b45d71b8ad P f1525cbc82a24d13b65f136695c903af reported cstate change: config changed from index -1 to 4, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:37.021628 641 ts_tablet_manager.cc:902] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47: Initiating tablet copy from peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:37.026876 641 tablet_copy_client.cc:308] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.129:43471
I20240501 08:12:37.028086 27781 tablet_copy_service.cc:140] P 6331f77dae9d452184e8cd0cbf5f1b43: Received BeginTabletCopySession request for tablet 802538381de843ab8d326014a0b4ec15 from peer 9a022e1a501f4d44a0468054d6c3de47 ({username='slave'} at 127.21.84.134:48929)
I20240501 08:12:37.028365 27781 tablet_copy_service.cc:161] P 6331f77dae9d452184e8cd0cbf5f1b43: Beginning new tablet copy session on tablet 802538381de843ab8d326014a0b4ec15 from peer 9a022e1a501f4d44a0468054d6c3de47 at {username='slave'} at 127.21.84.134:48929: session id = 9a022e1a501f4d44a0468054d6c3de47-802538381de843ab8d326014a0b4ec15
I20240501 08:12:37.032258 27781 tablet_copy_source_session.cc:205] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:37.033452 28021 consensus_queue.cc:237] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
I20240501 08:12:37.035679 641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 802538381de843ab8d326014a0b4ec15. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 08:12:37.037037 27956 consensus_peers.cc:485] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 -> Peer 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): Couldn't send request to peer 64998323a1f4482d8704bf20594d13a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5eccb9416d7d438893307f13dd4e73dc. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:37.037567 27761 raft_consensus.cc:1261] T 5eccb9416d7d438893307f13dd4e73dc P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:37.037568 28150 raft_consensus.cc:1261] T 5eccb9416d7d438893307f13dd4e73dc P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:37.038136 32641 consensus_queue.cc:1035] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:37.038882 32641 consensus_queue.cc:1035] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:37.041666 32641 raft_consensus.cc:2932] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:12:37.042593 641 tablet_copy_client.cc:783] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:37.042865 641 tablet_copy_client.cc:647] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:37.046322 27761 raft_consensus.cc:2932] T 5eccb9416d7d438893307f13dd4e73dc P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:12:37.049882 641 tablet_copy_client.cc:523] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:37.052379 27615 catalog_manager.cc:5484] T 5eccb9416d7d438893307f13dd4e73dc P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index -1 to 4, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:12:37.054842 641 tablet_bootstrap.cc:492] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:37.063709 656 ts_tablet_manager.cc:902] T 00a73d23b12d48a2966f94789db4369a P 91d11eae5ef6432ca86eb8ed446b6ea6: Initiating tablet copy from peer 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:37.080401 28150 raft_consensus.cc:2932] T 5eccb9416d7d438893307f13dd4e73dc P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:12:37.096079 28150 consensus_queue.cc:237] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
W20240501 08:12:37.108902 28088 consensus_peers.cc:485] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6 -> Peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): Couldn't send request to peer 31811708c6774037b82df44024b229b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e40fb0a0d49f44a1a36c33412c630077. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:37.109635 27889 raft_consensus.cc:1261] T e40fb0a0d49f44a1a36c33412c630077 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:37.110124 28021 raft_consensus.cc:1261] T e40fb0a0d49f44a1a36c33412c630077 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:37.110160 584 consensus_queue.cc:1035] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:37.110661 32293 consensus_queue.cc:1035] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:37.115454 32293 raft_consensus.cc:2932] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:12:37.127854 28021 raft_consensus.cc:2932] T e40fb0a0d49f44a1a36c33412c630077 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:12:37.129971 27889 raft_consensus.cc:2932] T e40fb0a0d49f44a1a36c33412c630077 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:12:37.136044 27619 catalog_manager.cc:5484] T e40fb0a0d49f44a1a36c33412c630077 P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index -1 to 4, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:12:37.137928 641 tablet_bootstrap.cc:492] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:37.138528 641 tablet_bootstrap.cc:492] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap complete.
I20240501 08:12:37.138849 641 ts_tablet_manager.cc:1367] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.084s user 0.054s sys 0.013s
I20240501 08:12:37.139802 641 raft_consensus.cc:348] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:12:37.140092 641 raft_consensus.cc:729] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: LEARNER
I20240501 08:12:37.140319 641 consensus_queue.cc:260] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:12:37.140926 641 ts_tablet_manager.cc:1398] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:37.141835 27781 tablet_copy_service.cc:342] P 6331f77dae9d452184e8cd0cbf5f1b43: Request end of tablet copy session 9a022e1a501f4d44a0468054d6c3de47-802538381de843ab8d326014a0b4ec15 received from {username='slave'} at 127.21.84.134:48929
I20240501 08:12:37.142009 27781 tablet_copy_service.cc:433] P 6331f77dae9d452184e8cd0cbf5f1b43: ending tablet copy session 9a022e1a501f4d44a0468054d6c3de47-802538381de843ab8d326014a0b4ec15 on tablet 802538381de843ab8d326014a0b4ec15 with peer 9a022e1a501f4d44a0468054d6c3de47
I20240501 08:12:37.144850 656 tablet_copy_client.cc:308] T 00a73d23b12d48a2966f94789db4369a P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.135:33503
I20240501 08:12:37.147138 30825 tablet_copy_service.cc:140] P 64998323a1f4482d8704bf20594d13a5: Received BeginTabletCopySession request for tablet 00a73d23b12d48a2966f94789db4369a from peer 91d11eae5ef6432ca86eb8ed446b6ea6 ({username='slave'} at 127.21.84.132:54591)
I20240501 08:12:37.147384 30825 tablet_copy_service.cc:161] P 64998323a1f4482d8704bf20594d13a5: Beginning new tablet copy session on tablet 00a73d23b12d48a2966f94789db4369a from peer 91d11eae5ef6432ca86eb8ed446b6ea6 at {username='slave'} at 127.21.84.132:54591: session id = 91d11eae5ef6432ca86eb8ed446b6ea6-00a73d23b12d48a2966f94789db4369a
I20240501 08:12:37.160198 30825 tablet_copy_source_session.cc:205] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:37.163661 656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00a73d23b12d48a2966f94789db4369a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:37.176673 656 tablet_copy_client.cc:783] T 00a73d23b12d48a2966f94789db4369a P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:37.177309 656 tablet_copy_client.cc:647] T 00a73d23b12d48a2966f94789db4369a P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:37.180327 656 tablet_copy_client.cc:523] T 00a73d23b12d48a2966f94789db4369a P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:37.186933 672 ts_tablet_manager.cc:902] T 683aa9595f2e456c943fc4d16f22e83c P 6331f77dae9d452184e8cd0cbf5f1b43: Initiating tablet copy from peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:37.189687 656 tablet_bootstrap.cc:492] T 00a73d23b12d48a2966f94789db4369a P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:37.198211 675 ts_tablet_manager.cc:902] T 131eac1e19264e00b76500d2ba42697a P f1525cbc82a24d13b65f136695c903af: Initiating tablet copy from peer 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:37.201735 656 tablet_bootstrap.cc:492] T 00a73d23b12d48a2966f94789db4369a P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:37.203043 656 tablet_bootstrap.cc:492] T 00a73d23b12d48a2966f94789db4369a P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap complete.
I20240501 08:12:37.203374 656 ts_tablet_manager.cc:1367] T 00a73d23b12d48a2966f94789db4369a P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20240501 08:12:37.204420 656 raft_consensus.cc:348] T 00a73d23b12d48a2966f94789db4369a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:12:37.204932 656 raft_consensus.cc:729] T 00a73d23b12d48a2966f94789db4369a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: LEARNER
I20240501 08:12:37.205200 656 consensus_queue.cc:260] T 00a73d23b12d48a2966f94789db4369a P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:12:37.206166 656 ts_tablet_manager.cc:1398] T 00a73d23b12d48a2966f94789db4369a P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:12:37.207648 30825 tablet_copy_service.cc:342] P 64998323a1f4482d8704bf20594d13a5: Request end of tablet copy session 91d11eae5ef6432ca86eb8ed446b6ea6-00a73d23b12d48a2966f94789db4369a received from {username='slave'} at 127.21.84.132:54591
I20240501 08:12:37.207819 30825 tablet_copy_service.cc:433] P 64998323a1f4482d8704bf20594d13a5: ending tablet copy session 91d11eae5ef6432ca86eb8ed446b6ea6-00a73d23b12d48a2966f94789db4369a on tablet 00a73d23b12d48a2966f94789db4369a with peer 91d11eae5ef6432ca86eb8ed446b6ea6
I20240501 08:12:37.237908 28021 consensus_queue.cc:237] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:12:37.246394 672 tablet_copy_client.cc:308] T 683aa9595f2e456c943fc4d16f22e83c P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.134:39933
I20240501 08:12:37.249177 675 tablet_copy_client.cc:308] T 131eac1e19264e00b76500d2ba42697a P f1525cbc82a24d13b65f136695c903af: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.135:33503
W20240501 08:12:37.251046 27957 consensus_peers.cc:485] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 -> Peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): Couldn't send request to peer a29cd590999b4f32849ff5689cd53c11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b12559fc416a4f3cbffa018611226bcc. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:37.251245 30825 tablet_copy_service.cc:140] P 64998323a1f4482d8704bf20594d13a5: Received BeginTabletCopySession request for tablet 131eac1e19264e00b76500d2ba42697a from peer f1525cbc82a24d13b65f136695c903af ({username='slave'} at 127.21.84.130:45079)
I20240501 08:12:37.251412 30825 tablet_copy_service.cc:161] P 64998323a1f4482d8704bf20594d13a5: Beginning new tablet copy session on tablet 131eac1e19264e00b76500d2ba42697a from peer f1525cbc82a24d13b65f136695c903af at {username='slave'} at 127.21.84.130:45079: session id = f1525cbc82a24d13b65f136695c903af-131eac1e19264e00b76500d2ba42697a
I20240501 08:12:37.252403 28150 raft_consensus.cc:1261] T b12559fc416a4f3cbffa018611226bcc P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:37.252931 32641 consensus_queue.cc:1035] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:37.256539 27889 raft_consensus.cc:1261] T b12559fc416a4f3cbffa018611226bcc P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:37.257200 32641 consensus_queue.cc:1035] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:37.253912 30825 tablet_copy_source_session.cc:205] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:37.263360 675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 131eac1e19264e00b76500d2ba42697a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:37.263819 419 raft_consensus.cc:2932] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:37.266211 30693 tablet_copy_service.cc:140] P 9a022e1a501f4d44a0468054d6c3de47: Received BeginTabletCopySession request for tablet 683aa9595f2e456c943fc4d16f22e83c from peer 6331f77dae9d452184e8cd0cbf5f1b43 ({username='slave'} at 127.21.84.129:59569)
I20240501 08:12:37.266454 30693 tablet_copy_service.cc:161] P 9a022e1a501f4d44a0468054d6c3de47: Beginning new tablet copy session on tablet 683aa9595f2e456c943fc4d16f22e83c from peer 6331f77dae9d452184e8cd0cbf5f1b43 at {username='slave'} at 127.21.84.129:59569: session id = 6331f77dae9d452184e8cd0cbf5f1b43-683aa9595f2e456c943fc4d16f22e83c
I20240501 08:12:37.270299 28150 raft_consensus.cc:2932] T b12559fc416a4f3cbffa018611226bcc P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:37.273049 27619 catalog_manager.cc:5484] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index -1 to 4, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:37.280862 27890 raft_consensus.cc:2932] T b12559fc416a4f3cbffa018611226bcc P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:37.281586 30693 tablet_copy_source_session.cc:205] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:37.291791 675 tablet_copy_client.cc:783] T 131eac1e19264e00b76500d2ba42697a P f1525cbc82a24d13b65f136695c903af: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:37.292116 675 tablet_copy_client.cc:647] T 131eac1e19264e00b76500d2ba42697a P f1525cbc82a24d13b65f136695c903af: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:37.297658 675 tablet_copy_client.cc:523] T 131eac1e19264e00b76500d2ba42697a P f1525cbc82a24d13b65f136695c903af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:37.299571 691 ts_tablet_manager.cc:902] T 103e1394bfe34a86ba1d9487d75fd512 P f1525cbc82a24d13b65f136695c903af: Initiating tablet copy from peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:37.304411 672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 683aa9595f2e456c943fc4d16f22e83c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:37.311906 656 ts_tablet_manager.cc:902] T 4fd595c79cdc43f9a8afadb6b548bf64 P 91d11eae5ef6432ca86eb8ed446b6ea6: Initiating tablet copy from peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:12:37.312515 675 tablet_bootstrap.cc:492] T 131eac1e19264e00b76500d2ba42697a P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:37.322454 691 tablet_copy_client.cc:308] T 103e1394bfe34a86ba1d9487d75fd512 P f1525cbc82a24d13b65f136695c903af: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.129:43471
I20240501 08:12:37.323570 27781 tablet_copy_service.cc:140] P 6331f77dae9d452184e8cd0cbf5f1b43: Received BeginTabletCopySession request for tablet 103e1394bfe34a86ba1d9487d75fd512 from peer f1525cbc82a24d13b65f136695c903af ({username='slave'} at 127.21.84.130:44301)
I20240501 08:12:37.323762 27781 tablet_copy_service.cc:161] P 6331f77dae9d452184e8cd0cbf5f1b43: Beginning new tablet copy session on tablet 103e1394bfe34a86ba1d9487d75fd512 from peer f1525cbc82a24d13b65f136695c903af at {username='slave'} at 127.21.84.130:44301: session id = f1525cbc82a24d13b65f136695c903af-103e1394bfe34a86ba1d9487d75fd512
I20240501 08:12:37.325635 675 tablet_bootstrap.cc:492] T 131eac1e19264e00b76500d2ba42697a P f1525cbc82a24d13b65f136695c903af: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:37.326085 675 tablet_bootstrap.cc:492] T 131eac1e19264e00b76500d2ba42697a P f1525cbc82a24d13b65f136695c903af: Bootstrap complete.
I20240501 08:12:37.326350 675 ts_tablet_manager.cc:1367] T 131eac1e19264e00b76500d2ba42697a P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.008s
I20240501 08:12:37.327097 675 raft_consensus.cc:348] T 131eac1e19264e00b76500d2ba42697a P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:37.327296 27781 tablet_copy_source_session.cc:205] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:37.327351 675 raft_consensus.cc:729] T 131eac1e19264e00b76500d2ba42697a P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: LEARNER
I20240501 08:12:37.327561 675 consensus_queue.cc:260] T 131eac1e19264e00b76500d2ba42697a P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:37.328112 675 ts_tablet_manager.cc:1398] T 131eac1e19264e00b76500d2ba42697a P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:37.329430 691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 103e1394bfe34a86ba1d9487d75fd512. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:37.331358 672 tablet_copy_client.cc:783] T 683aa9595f2e456c943fc4d16f22e83c P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:37.331630 672 tablet_copy_client.cc:647] T 683aa9595f2e456c943fc4d16f22e83c P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:37.333917 672 tablet_copy_client.cc:523] T 683aa9595f2e456c943fc4d16f22e83c P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:37.334509 30825 tablet_copy_service.cc:342] P 64998323a1f4482d8704bf20594d13a5: Request end of tablet copy session f1525cbc82a24d13b65f136695c903af-131eac1e19264e00b76500d2ba42697a received from {username='slave'} at 127.21.84.130:45079
I20240501 08:12:37.334688 30825 tablet_copy_service.cc:433] P 64998323a1f4482d8704bf20594d13a5: ending tablet copy session f1525cbc82a24d13b65f136695c903af-131eac1e19264e00b76500d2ba42697a on tablet 131eac1e19264e00b76500d2ba42697a with peer f1525cbc82a24d13b65f136695c903af
I20240501 08:12:37.339952 672 tablet_bootstrap.cc:492] T 683aa9595f2e456c943fc4d16f22e83c P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:37.341403 691 tablet_copy_client.cc:783] T 103e1394bfe34a86ba1d9487d75fd512 P f1525cbc82a24d13b65f136695c903af: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:37.341522 641 ts_tablet_manager.cc:902] T 946add184f4a46f28fd631e8a3bc0ba7 P 9a022e1a501f4d44a0468054d6c3de47: Initiating tablet copy from peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:37.341619 691 tablet_copy_client.cc:647] T 103e1394bfe34a86ba1d9487d75fd512 P f1525cbc82a24d13b65f136695c903af: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:37.344956 691 tablet_copy_client.cc:523] T 103e1394bfe34a86ba1d9487d75fd512 P f1525cbc82a24d13b65f136695c903af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:37.354048 27889 consensus_queue.cc:237] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
W20240501 08:12:37.359196 27827 consensus_peers.cc:485] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af -> Peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Couldn't send request to peer aad81047009a4ffcbc450b60030583b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d6bb70263134414b9945a87087b55a2c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:37.359402 691 tablet_bootstrap.cc:492] T 103e1394bfe34a86ba1d9487d75fd512 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:37.360407 30537 raft_consensus.cc:1261] T d6bb70263134414b9945a87087b55a2c P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:37.361707 32079 consensus_queue.cc:1035] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:37.368633 672 tablet_bootstrap.cc:492] T 683aa9595f2e456c943fc4d16f22e83c P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:37.369302 672 tablet_bootstrap.cc:492] T 683aa9595f2e456c943fc4d16f22e83c P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap complete.
I20240501 08:12:37.369768 672 ts_tablet_manager.cc:1367] T 683aa9595f2e456c943fc4d16f22e83c P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.011s
I20240501 08:12:37.371088 672 raft_consensus.cc:348] T 683aa9595f2e456c943fc4d16f22e83c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:12:37.371404 672 raft_consensus.cc:729] T 683aa9595f2e456c943fc4d16f22e83c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: LEARNER
I20240501 08:12:37.371667 672 consensus_queue.cc:260] T 683aa9595f2e456c943fc4d16f22e83c P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:12:37.375741 30671 raft_consensus.cc:1261] T d6bb70263134414b9945a87087b55a2c P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:37.376843 596 consensus_queue.cc:1035] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:37.390663 672 ts_tablet_manager.cc:1398] T 683aa9595f2e456c943fc4d16f22e83c P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.021s user 0.000s sys 0.002s
I20240501 08:12:37.397325 656 tablet_copy_client.cc:308] T 4fd595c79cdc43f9a8afadb6b548bf64 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.130:36503
I20240501 08:12:37.397708 30693 tablet_copy_service.cc:342] P 9a022e1a501f4d44a0468054d6c3de47: Request end of tablet copy session 6331f77dae9d452184e8cd0cbf5f1b43-683aa9595f2e456c943fc4d16f22e83c received from {username='slave'} at 127.21.84.129:59569
I20240501 08:12:37.397898 30693 tablet_copy_service.cc:433] P 9a022e1a501f4d44a0468054d6c3de47: ending tablet copy session 6331f77dae9d452184e8cd0cbf5f1b43-683aa9595f2e456c943fc4d16f22e83c on tablet 683aa9595f2e456c943fc4d16f22e83c with peer 6331f77dae9d452184e8cd0cbf5f1b43
I20240501 08:12:37.398169 27912 tablet_copy_service.cc:140] P f1525cbc82a24d13b65f136695c903af: Received BeginTabletCopySession request for tablet 4fd595c79cdc43f9a8afadb6b548bf64 from peer 91d11eae5ef6432ca86eb8ed446b6ea6 ({username='slave'} at 127.21.84.132:35465)
I20240501 08:12:37.398339 27912 tablet_copy_service.cc:161] P f1525cbc82a24d13b65f136695c903af: Beginning new tablet copy session on tablet 4fd595c79cdc43f9a8afadb6b548bf64 from peer 91d11eae5ef6432ca86eb8ed446b6ea6 at {username='slave'} at 127.21.84.132:35465: session id = 91d11eae5ef6432ca86eb8ed446b6ea6-4fd595c79cdc43f9a8afadb6b548bf64
I20240501 08:12:37.407694 27912 tablet_copy_source_session.cc:205] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:37.420658 714 ts_tablet_manager.cc:902] T 8ab0410a98974002ab68dd0f6a456d59 P 64998323a1f4482d8704bf20594d13a5: Initiating tablet copy from peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:37.422974 656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fd595c79cdc43f9a8afadb6b548bf64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:37.429191 715 ts_tablet_manager.cc:902] T c4851258508d46c88493503a6257f365 P 91d11eae5ef6432ca86eb8ed446b6ea6: Initiating tablet copy from peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:12:37.429677 641 tablet_copy_client.cc:308] T 946add184f4a46f28fd631e8a3bc0ba7 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.136:46845
I20240501 08:12:37.430678 30957 tablet_copy_service.cc:140] P a29cd590999b4f32849ff5689cd53c11: Received BeginTabletCopySession request for tablet 946add184f4a46f28fd631e8a3bc0ba7 from peer 9a022e1a501f4d44a0468054d6c3de47 ({username='slave'} at 127.21.84.134:36153)
I20240501 08:12:37.430908 30957 tablet_copy_service.cc:161] P a29cd590999b4f32849ff5689cd53c11: Beginning new tablet copy session on tablet 946add184f4a46f28fd631e8a3bc0ba7 from peer 9a022e1a501f4d44a0468054d6c3de47 at {username='slave'} at 127.21.84.134:36153: session id = 9a022e1a501f4d44a0468054d6c3de47-946add184f4a46f28fd631e8a3bc0ba7
I20240501 08:12:37.432523 596 raft_consensus.cc:2932] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:37.434432 30957 tablet_copy_source_session.cc:205] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:37.437044 641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 946add184f4a46f28fd631e8a3bc0ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:37.438488 656 tablet_copy_client.cc:783] T 4fd595c79cdc43f9a8afadb6b548bf64 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:37.438709 656 tablet_copy_client.cc:647] T 4fd595c79cdc43f9a8afadb6b548bf64 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:37.438330 30537 raft_consensus.cc:2932] T d6bb70263134414b9945a87087b55a2c P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:37.444456 30671 raft_consensus.cc:2932] T d6bb70263134414b9945a87087b55a2c P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:37.447620 30669 raft_consensus.cc:1203] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:37.451431 27612 catalog_manager.cc:5484] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af reported cstate change: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:37.462323 656 tablet_copy_client.cc:523] T 4fd595c79cdc43f9a8afadb6b548bf64 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:37.465163 641 tablet_copy_client.cc:783] T 946add184f4a46f28fd631e8a3bc0ba7 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:37.465412 641 tablet_copy_client.cc:647] T 946add184f4a46f28fd631e8a3bc0ba7 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:37.466825 656 tablet_bootstrap.cc:492] T 4fd595c79cdc43f9a8afadb6b548bf64 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:37.468214 691 tablet_bootstrap.cc:492] T 103e1394bfe34a86ba1d9487d75fd512 P f1525cbc82a24d13b65f136695c903af: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:37.468591 691 tablet_bootstrap.cc:492] T 103e1394bfe34a86ba1d9487d75fd512 P f1525cbc82a24d13b65f136695c903af: Bootstrap complete.
I20240501 08:12:37.474284 691 ts_tablet_manager.cc:1367] T 103e1394bfe34a86ba1d9487d75fd512 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.115s user 0.019s sys 0.054s
I20240501 08:12:37.475138 691 raft_consensus.cc:348] T 103e1394bfe34a86ba1d9487d75fd512 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:37.475363 691 raft_consensus.cc:729] T 103e1394bfe34a86ba1d9487d75fd512 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: LEARNER
I20240501 08:12:37.475569 691 consensus_queue.cc:260] T 103e1394bfe34a86ba1d9487d75fd512 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:37.476159 691 ts_tablet_manager.cc:1398] T 103e1394bfe34a86ba1d9487d75fd512 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:37.483117 27781 tablet_copy_service.cc:342] P 6331f77dae9d452184e8cd0cbf5f1b43: Request end of tablet copy session f1525cbc82a24d13b65f136695c903af-103e1394bfe34a86ba1d9487d75fd512 received from {username='slave'} at 127.21.84.130:44301
I20240501 08:12:37.483242 27781 tablet_copy_service.cc:433] P 6331f77dae9d452184e8cd0cbf5f1b43: ending tablet copy session f1525cbc82a24d13b65f136695c903af-103e1394bfe34a86ba1d9487d75fd512 on tablet 103e1394bfe34a86ba1d9487d75fd512 with peer f1525cbc82a24d13b65f136695c903af
I20240501 08:12:37.483729 641 tablet_copy_client.cc:523] T 946add184f4a46f28fd631e8a3bc0ba7 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:37.484679 721 ts_tablet_manager.cc:902] T 5eccb9416d7d438893307f13dd4e73dc P 64998323a1f4482d8704bf20594d13a5: Initiating tablet copy from peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:37.488138 641 tablet_bootstrap.cc:492] T 946add184f4a46f28fd631e8a3bc0ba7 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:37.497051 656 tablet_bootstrap.cc:492] T 4fd595c79cdc43f9a8afadb6b548bf64 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:37.497422 656 tablet_bootstrap.cc:492] T 4fd595c79cdc43f9a8afadb6b548bf64 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap complete.
I20240501 08:12:37.497658 656 ts_tablet_manager.cc:1367] T 4fd595c79cdc43f9a8afadb6b548bf64 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.007s
I20240501 08:12:37.498416 656 raft_consensus.cc:348] T 4fd595c79cdc43f9a8afadb6b548bf64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:12:37.498625 656 raft_consensus.cc:729] T 4fd595c79cdc43f9a8afadb6b548bf64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: LEARNER
I20240501 08:12:37.498771 656 consensus_queue.cc:260] T 4fd595c79cdc43f9a8afadb6b548bf64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:12:37.499290 656 ts_tablet_manager.cc:1398] T 4fd595c79cdc43f9a8afadb6b548bf64 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:12:37.501328 27912 tablet_copy_service.cc:342] P f1525cbc82a24d13b65f136695c903af: Request end of tablet copy session 91d11eae5ef6432ca86eb8ed446b6ea6-4fd595c79cdc43f9a8afadb6b548bf64 received from {username='slave'} at 127.21.84.132:35465
I20240501 08:12:37.501451 27912 tablet_copy_service.cc:433] P f1525cbc82a24d13b65f136695c903af: ending tablet copy session 91d11eae5ef6432ca86eb8ed446b6ea6-4fd595c79cdc43f9a8afadb6b548bf64 on tablet 4fd595c79cdc43f9a8afadb6b548bf64 with peer 91d11eae5ef6432ca86eb8ed446b6ea6
I20240501 08:12:37.510082 641 tablet_bootstrap.cc:492] T 946add184f4a46f28fd631e8a3bc0ba7 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:37.511898 721 tablet_copy_client.cc:308] T 5eccb9416d7d438893307f13dd4e73dc P 64998323a1f4482d8704bf20594d13a5: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.131:36305
I20240501 08:12:37.512897 28042 tablet_copy_service.cc:140] P aad81047009a4ffcbc450b60030583b5: Received BeginTabletCopySession request for tablet 5eccb9416d7d438893307f13dd4e73dc from peer 64998323a1f4482d8704bf20594d13a5 ({username='slave'} at 127.21.84.135:55053)
I20240501 08:12:37.513152 28042 tablet_copy_service.cc:161] P aad81047009a4ffcbc450b60030583b5: Beginning new tablet copy session on tablet 5eccb9416d7d438893307f13dd4e73dc from peer 64998323a1f4482d8704bf20594d13a5 at {username='slave'} at 127.21.84.135:55053: session id = 64998323a1f4482d8704bf20594d13a5-5eccb9416d7d438893307f13dd4e73dc
I20240501 08:12:37.513612 641 tablet_bootstrap.cc:492] T 946add184f4a46f28fd631e8a3bc0ba7 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap complete.
I20240501 08:12:37.514024 641 ts_tablet_manager.cc:1367] T 946add184f4a46f28fd631e8a3bc0ba7 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.001s
I20240501 08:12:37.515141 715 tablet_copy_client.cc:308] T c4851258508d46c88493503a6257f365 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.130:36503
I20240501 08:12:37.516731 27912 tablet_copy_service.cc:140] P f1525cbc82a24d13b65f136695c903af: Received BeginTabletCopySession request for tablet c4851258508d46c88493503a6257f365 from peer 91d11eae5ef6432ca86eb8ed446b6ea6 ({username='slave'} at 127.21.84.132:35465)
I20240501 08:12:37.516870 27912 tablet_copy_service.cc:161] P f1525cbc82a24d13b65f136695c903af: Beginning new tablet copy session on tablet c4851258508d46c88493503a6257f365 from peer 91d11eae5ef6432ca86eb8ed446b6ea6 at {username='slave'} at 127.21.84.132:35465: session id = 91d11eae5ef6432ca86eb8ed446b6ea6-c4851258508d46c88493503a6257f365
I20240501 08:12:37.519541 27912 tablet_copy_source_session.cc:205] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:37.517087 641 raft_consensus.cc:348] T 946add184f4a46f28fd631e8a3bc0ba7 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:12:37.520004 641 raft_consensus.cc:729] T 946add184f4a46f28fd631e8a3bc0ba7 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: LEARNER
I20240501 08:12:37.521464 715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4851258508d46c88493503a6257f365. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:37.520210 641 consensus_queue.cc:260] T 946add184f4a46f28fd631e8a3bc0ba7 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:12:37.527654 641 ts_tablet_manager.cc:1398] T 946add184f4a46f28fd631e8a3bc0ba7 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240501 08:12:37.528515 714 tablet_copy_client.cc:308] T 8ab0410a98974002ab68dd0f6a456d59 P 64998323a1f4482d8704bf20594d13a5: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.129:43471
I20240501 08:12:37.528915 30957 tablet_copy_service.cc:342] P a29cd590999b4f32849ff5689cd53c11: Request end of tablet copy session 9a022e1a501f4d44a0468054d6c3de47-946add184f4a46f28fd631e8a3bc0ba7 received from {username='slave'} at 127.21.84.134:36153
I20240501 08:12:37.528940 28042 tablet_copy_source_session.cc:205] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:37.529093 30957 tablet_copy_service.cc:433] P a29cd590999b4f32849ff5689cd53c11: ending tablet copy session 9a022e1a501f4d44a0468054d6c3de47-946add184f4a46f28fd631e8a3bc0ba7 on tablet 946add184f4a46f28fd631e8a3bc0ba7 with peer 9a022e1a501f4d44a0468054d6c3de47
I20240501 08:12:37.530153 27781 tablet_copy_service.cc:140] P 6331f77dae9d452184e8cd0cbf5f1b43: Received BeginTabletCopySession request for tablet 8ab0410a98974002ab68dd0f6a456d59 from peer 64998323a1f4482d8704bf20594d13a5 ({username='slave'} at 127.21.84.135:52115)
I20240501 08:12:37.530277 27781 tablet_copy_service.cc:161] P 6331f77dae9d452184e8cd0cbf5f1b43: Beginning new tablet copy session on tablet 8ab0410a98974002ab68dd0f6a456d59 from peer 64998323a1f4482d8704bf20594d13a5 at {username='slave'} at 127.21.84.135:52115: session id = 64998323a1f4482d8704bf20594d13a5-8ab0410a98974002ab68dd0f6a456d59
I20240501 08:12:37.532307 715 tablet_copy_client.cc:783] T c4851258508d46c88493503a6257f365 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:37.532508 715 tablet_copy_client.cc:647] T c4851258508d46c88493503a6257f365 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:37.532984 27781 tablet_copy_source_session.cc:205] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:37.536235 28152 consensus_queue.cc:237] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:37.537045 714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab0410a98974002ab68dd0f6a456d59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:37.537416 721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eccb9416d7d438893307f13dd4e73dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:37.540773 715 tablet_copy_client.cc:523] T c4851258508d46c88493503a6257f365 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:37.544651 715 tablet_bootstrap.cc:492] T c4851258508d46c88493503a6257f365 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:37.547463 27759 raft_consensus.cc:1261] T 489db424748b4a73966b047e81f36c83 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:37.547840 30536 raft_consensus.cc:1261] T 489db424748b4a73966b047e81f36c83 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:37.548399 584 consensus_queue.cc:1035] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:37.549762 714 tablet_copy_client.cc:783] T 8ab0410a98974002ab68dd0f6a456d59 P 64998323a1f4482d8704bf20594d13a5: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:37.550149 714 tablet_copy_client.cc:647] T 8ab0410a98974002ab68dd0f6a456d59 P 64998323a1f4482d8704bf20594d13a5: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:37.554744 721 tablet_copy_client.cc:783] T 5eccb9416d7d438893307f13dd4e73dc P 64998323a1f4482d8704bf20594d13a5: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:37.555058 721 tablet_copy_client.cc:647] T 5eccb9416d7d438893307f13dd4e73dc P 64998323a1f4482d8704bf20594d13a5: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:37.556603 32293 consensus_queue.cc:1035] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:12:37.558465 28088 consensus_peers.cc:485] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 489db424748b4a73966b047e81f36c83. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:37.558506 721 tablet_copy_client.cc:523] T 5eccb9416d7d438893307f13dd4e73dc P 64998323a1f4482d8704bf20594d13a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:37.560369 714 tablet_copy_client.cc:523] T 8ab0410a98974002ab68dd0f6a456d59 P 64998323a1f4482d8704bf20594d13a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:37.564466 714 tablet_bootstrap.cc:492] T 8ab0410a98974002ab68dd0f6a456d59 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:37.572893 641 ts_tablet_manager.cc:902] T 5536362915034f3eb86138b45d71b8ad P 9a022e1a501f4d44a0468054d6c3de47: Initiating tablet copy from peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:37.582952 584 raft_consensus.cc:2932] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:37.584125 721 tablet_bootstrap.cc:492] T 5eccb9416d7d438893307f13dd4e73dc P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:37.588222 30536 raft_consensus.cc:2932] T 489db424748b4a73966b047e81f36c83 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:37.596467 27889 raft_consensus.cc:1203] T 131eac1e19264e00b76500d2ba42697a P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:37.596421 27619 catalog_manager.cc:5484] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:37.628504 27761 raft_consensus.cc:1203] T 683aa9595f2e456c943fc4d16f22e83c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:37.650058 28152 raft_consensus.cc:1203] T 00a73d23b12d48a2966f94789db4369a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:37.655139 641 tablet_copy_client.cc:308] T 5536362915034f3eb86138b45d71b8ad P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.132:41323
I20240501 08:12:37.656069 28172 tablet_copy_service.cc:140] P 91d11eae5ef6432ca86eb8ed446b6ea6: Received BeginTabletCopySession request for tablet 5536362915034f3eb86138b45d71b8ad from peer 9a022e1a501f4d44a0468054d6c3de47 ({username='slave'} at 127.21.84.134:40845)
I20240501 08:12:37.656354 28172 tablet_copy_service.cc:161] P 91d11eae5ef6432ca86eb8ed446b6ea6: Beginning new tablet copy session on tablet 5536362915034f3eb86138b45d71b8ad from peer 9a022e1a501f4d44a0468054d6c3de47 at {username='slave'} at 127.21.84.134:40845: session id = 9a022e1a501f4d44a0468054d6c3de47-5536362915034f3eb86138b45d71b8ad
I20240501 08:12:37.670612 28172 tablet_copy_source_session.cc:205] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:37.672955 641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5536362915034f3eb86138b45d71b8ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:37.673101 750 ts_tablet_manager.cc:902] T b12559fc416a4f3cbffa018611226bcc P a29cd590999b4f32849ff5689cd53c11: Initiating tablet copy from peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:37.673763 749 ts_tablet_manager.cc:902] T e40fb0a0d49f44a1a36c33412c630077 P 31811708c6774037b82df44024b229b6: Initiating tablet copy from peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:37.679647 714 tablet_bootstrap.cc:492] T 8ab0410a98974002ab68dd0f6a456d59 P 64998323a1f4482d8704bf20594d13a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:37.680253 714 tablet_bootstrap.cc:492] T 8ab0410a98974002ab68dd0f6a456d59 P 64998323a1f4482d8704bf20594d13a5: Bootstrap complete.
I20240501 08:12:37.680532 714 ts_tablet_manager.cc:1367] T 8ab0410a98974002ab68dd0f6a456d59 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.116s user 0.006s sys 0.006s
I20240501 08:12:37.683791 715 tablet_bootstrap.cc:492] T c4851258508d46c88493503a6257f365 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:37.684198 715 tablet_bootstrap.cc:492] T c4851258508d46c88493503a6257f365 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap complete.
I20240501 08:12:37.684437 715 ts_tablet_manager.cc:1367] T c4851258508d46c88493503a6257f365 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.140s user 0.050s sys 0.080s
I20240501 08:12:37.685164 715 raft_consensus.cc:348] T c4851258508d46c88493503a6257f365 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:12:37.685412 715 raft_consensus.cc:729] T c4851258508d46c88493503a6257f365 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: LEARNER
I20240501 08:12:37.685616 715 consensus_queue.cc:260] T c4851258508d46c88493503a6257f365 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:12:37.686200 641 tablet_copy_client.cc:783] T 5536362915034f3eb86138b45d71b8ad P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:37.686254 715 ts_tablet_manager.cc:1398] T c4851258508d46c88493503a6257f365 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 08:12:37.686829 27912 tablet_copy_service.cc:342] P f1525cbc82a24d13b65f136695c903af: Request end of tablet copy session 91d11eae5ef6432ca86eb8ed446b6ea6-c4851258508d46c88493503a6257f365 received from {username='slave'} at 127.21.84.132:35465
I20240501 08:12:37.686940 27912 tablet_copy_service.cc:433] P f1525cbc82a24d13b65f136695c903af: ending tablet copy session 91d11eae5ef6432ca86eb8ed446b6ea6-c4851258508d46c88493503a6257f365 on tablet c4851258508d46c88493503a6257f365 with peer 91d11eae5ef6432ca86eb8ed446b6ea6
I20240501 08:12:37.696574 641 tablet_copy_client.cc:647] T 5536362915034f3eb86138b45d71b8ad P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:37.697136 27759 raft_consensus.cc:2932] T 489db424748b4a73966b047e81f36c83 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:37.703817 641 tablet_copy_client.cc:523] T 5536362915034f3eb86138b45d71b8ad P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:37.707329 714 raft_consensus.cc:348] T 8ab0410a98974002ab68dd0f6a456d59 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
I20240501 08:12:37.707891 714 raft_consensus.cc:729] T 8ab0410a98974002ab68dd0f6a456d59 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: LEARNER
I20240501 08:12:37.708169 641 tablet_bootstrap.cc:492] T 5536362915034f3eb86138b45d71b8ad P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:37.708451 714 consensus_queue.cc:260] T 8ab0410a98974002ab68dd0f6a456d59 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
I20240501 08:12:37.727460 714 ts_tablet_manager.cc:1398] T 8ab0410a98974002ab68dd0f6a456d59 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.047s user 0.001s sys 0.002s
I20240501 08:12:37.728760 27781 tablet_copy_service.cc:342] P 6331f77dae9d452184e8cd0cbf5f1b43: Request end of tablet copy session 64998323a1f4482d8704bf20594d13a5-8ab0410a98974002ab68dd0f6a456d59 received from {username='slave'} at 127.21.84.135:52115
I20240501 08:12:37.728893 27781 tablet_copy_service.cc:433] P 6331f77dae9d452184e8cd0cbf5f1b43: ending tablet copy session 64998323a1f4482d8704bf20594d13a5-8ab0410a98974002ab68dd0f6a456d59 on tablet 8ab0410a98974002ab68dd0f6a456d59 with peer 64998323a1f4482d8704bf20594d13a5
I20240501 08:12:37.744601 27889 raft_consensus.cc:1203] T 103e1394bfe34a86ba1d9487d75fd512 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:37.754266 749 tablet_copy_client.cc:308] T e40fb0a0d49f44a1a36c33412c630077 P 31811708c6774037b82df44024b229b6: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.132:41323
I20240501 08:12:37.755895 28172 tablet_copy_service.cc:140] P 91d11eae5ef6432ca86eb8ed446b6ea6: Received BeginTabletCopySession request for tablet e40fb0a0d49f44a1a36c33412c630077 from peer 31811708c6774037b82df44024b229b6 ({username='slave'} at 127.21.84.133:45793)
I20240501 08:12:37.756078 28172 tablet_copy_service.cc:161] P 91d11eae5ef6432ca86eb8ed446b6ea6: Beginning new tablet copy session on tablet e40fb0a0d49f44a1a36c33412c630077 from peer 31811708c6774037b82df44024b229b6 at {username='slave'} at 127.21.84.133:45793: session id = 31811708c6774037b82df44024b229b6-e40fb0a0d49f44a1a36c33412c630077
I20240501 08:12:37.759250 28172 tablet_copy_source_session.cc:205] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:37.761765 749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e40fb0a0d49f44a1a36c33412c630077. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:37.772289 750 tablet_copy_client.cc:308] T b12559fc416a4f3cbffa018611226bcc P a29cd590999b4f32849ff5689cd53c11: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.131:36305
I20240501 08:12:37.773135 28042 tablet_copy_service.cc:140] P aad81047009a4ffcbc450b60030583b5: Received BeginTabletCopySession request for tablet b12559fc416a4f3cbffa018611226bcc from peer a29cd590999b4f32849ff5689cd53c11 ({username='slave'} at 127.21.84.136:51325)
I20240501 08:12:37.773288 28042 tablet_copy_service.cc:161] P aad81047009a4ffcbc450b60030583b5: Beginning new tablet copy session on tablet b12559fc416a4f3cbffa018611226bcc from peer a29cd590999b4f32849ff5689cd53c11 at {username='slave'} at 127.21.84.136:51325: session id = a29cd590999b4f32849ff5689cd53c11-b12559fc416a4f3cbffa018611226bcc
I20240501 08:12:37.776315 28042 tablet_copy_source_session.cc:205] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:37.777529 30669 raft_consensus.cc:1203] T 946add184f4a46f28fd631e8a3bc0ba7 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:37.778210 750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b12559fc416a4f3cbffa018611226bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:37.788448 750 tablet_copy_client.cc:783] T b12559fc416a4f3cbffa018611226bcc P a29cd590999b4f32849ff5689cd53c11: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:37.788780 750 tablet_copy_client.cc:647] T b12559fc416a4f3cbffa018611226bcc P a29cd590999b4f32849ff5689cd53c11: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:37.788920 765 ts_tablet_manager.cc:902] T d6bb70263134414b9945a87087b55a2c P aad81047009a4ffcbc450b60030583b5: Initiating tablet copy from peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:12:37.789036 30935 consensus_queue.cc:237] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:12:37.793393 750 tablet_copy_client.cc:523] T b12559fc416a4f3cbffa018611226bcc P a29cd590999b4f32849ff5689cd53c11: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:37.797369 750 tablet_bootstrap.cc:492] T b12559fc416a4f3cbffa018611226bcc P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:37.806944 641 tablet_bootstrap.cc:492] T 5536362915034f3eb86138b45d71b8ad P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:37.807338 749 tablet_copy_client.cc:783] T e40fb0a0d49f44a1a36c33412c630077 P 31811708c6774037b82df44024b229b6: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:37.807813 641 tablet_bootstrap.cc:492] T 5536362915034f3eb86138b45d71b8ad P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap complete.
I20240501 08:12:37.808225 749 tablet_copy_client.cc:647] T e40fb0a0d49f44a1a36c33412c630077 P 31811708c6774037b82df44024b229b6: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:37.808509 641 ts_tablet_manager.cc:1367] T 5536362915034f3eb86138b45d71b8ad P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.100s user 0.049s sys 0.003s
I20240501 08:12:37.810326 30803 raft_consensus.cc:1261] T 6c324b1136ce429290d7286399bb7121 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:37.810961 641 raft_consensus.cc:348] T 5536362915034f3eb86138b45d71b8ad P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:12:37.811240 641 raft_consensus.cc:729] T 5536362915034f3eb86138b45d71b8ad P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: LEARNER
I20240501 08:12:37.811539 641 consensus_queue.cc:260] T 5536362915034f3eb86138b45d71b8ad P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:12:37.812045 456 consensus_queue.cc:1035] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:37.816155 641 ts_tablet_manager.cc:1398] T 5536362915034f3eb86138b45d71b8ad P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 08:12:37.817082 28172 tablet_copy_service.cc:342] P 91d11eae5ef6432ca86eb8ed446b6ea6: Request end of tablet copy session 9a022e1a501f4d44a0468054d6c3de47-5536362915034f3eb86138b45d71b8ad received from {username='slave'} at 127.21.84.134:40845
I20240501 08:12:37.817235 28172 tablet_copy_service.cc:433] P 91d11eae5ef6432ca86eb8ed446b6ea6: ending tablet copy session 9a022e1a501f4d44a0468054d6c3de47-5536362915034f3eb86138b45d71b8ad on tablet 5536362915034f3eb86138b45d71b8ad with peer 9a022e1a501f4d44a0468054d6c3de47
I20240501 08:12:37.822209 30669 raft_consensus.cc:1261] T 6c324b1136ce429290d7286399bb7121 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:12:37.823196 30870 consensus_peers.cc:485] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 -> Peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Couldn't send request to peer 91d11eae5ef6432ca86eb8ed446b6ea6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c324b1136ce429290d7286399bb7121. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:37.824226 32634 consensus_queue.cc:1035] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:37.827611 749 tablet_copy_client.cc:523] T e40fb0a0d49f44a1a36c33412c630077 P 31811708c6774037b82df44024b229b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:37.828557 28152 raft_consensus.cc:1203] T c4851258508d46c88493503a6257f365 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:37.839603 749 tablet_bootstrap.cc:492] T e40fb0a0d49f44a1a36c33412c630077 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:37.865305 456 raft_consensus.cc:2932] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:12:37.878937 27619 catalog_manager.cc:5484] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:37.880784 30803 raft_consensus.cc:2932] T 6c324b1136ce429290d7286399bb7121 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:12:37.897331 30669 raft_consensus.cc:2932] T 6c324b1136ce429290d7286399bb7121 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:12:37.900851 28150 raft_consensus.cc:1203] T 4fd595c79cdc43f9a8afadb6b548bf64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240501 08:12:37.912113 30470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.132:41323, user_credentials={real_user=slave}} blocked reactor thread for 56679.5us
I20240501 08:12:37.961711 30803 raft_consensus.cc:1203] T 8ab0410a98974002ab68dd0f6a456d59 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:38.008266 750 tablet_bootstrap.cc:492] T b12559fc416a4f3cbffa018611226bcc P a29cd590999b4f32849ff5689cd53c11: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:38.012771 563 raft_consensus.cc:1050] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43: attempting to promote NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 to VOTER
I20240501 08:12:38.015022 750 tablet_bootstrap.cc:492] T b12559fc416a4f3cbffa018611226bcc P a29cd590999b4f32849ff5689cd53c11: Bootstrap complete.
I20240501 08:12:38.016052 750 ts_tablet_manager.cc:1367] T b12559fc416a4f3cbffa018611226bcc P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.219s user 0.061s sys 0.013s
I20240501 08:12:38.016336 563 consensus_queue.cc:237] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:12:38.021723 750 raft_consensus.cc:348] T b12559fc416a4f3cbffa018611226bcc P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:12:38.022058 750 raft_consensus.cc:729] T b12559fc416a4f3cbffa018611226bcc P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: LEARNER
I20240501 08:12:38.022264 750 consensus_queue.cc:260] T b12559fc416a4f3cbffa018611226bcc P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:12:38.023963 750 ts_tablet_manager.cc:1398] T b12559fc416a4f3cbffa018611226bcc P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 08:12:38.025353 28042 tablet_copy_service.cc:342] P aad81047009a4ffcbc450b60030583b5: Request end of tablet copy session a29cd590999b4f32849ff5689cd53c11-b12559fc416a4f3cbffa018611226bcc received from {username='slave'} at 127.21.84.136:51325
I20240501 08:12:38.025529 28042 tablet_copy_service.cc:433] P aad81047009a4ffcbc450b60030583b5: ending tablet copy session a29cd590999b4f32849ff5689cd53c11-b12559fc416a4f3cbffa018611226bcc on tablet b12559fc416a4f3cbffa018611226bcc with peer a29cd590999b4f32849ff5689cd53c11
I20240501 08:12:38.048945 30670 raft_consensus.cc:1261] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:38.049558 312 consensus_queue.cc:1035] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:38.053628 28022 raft_consensus.cc:1261] T 802538381de843ab8d326014a0b4ec15 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:38.054945 623 consensus_queue.cc:1035] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:38.069933 27889 raft_consensus.cc:1261] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:38.071439 623 consensus_queue.cc:1035] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:38.085932 312 raft_consensus.cc:2932] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:38.090220 32302 raft_consensus.cc:1050] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5: attempting to promote NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 to VOTER
I20240501 08:12:38.091485 32302 consensus_queue.cc:237] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:12:38.093057 30670 raft_consensus.cc:2932] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:38.097687 28022 raft_consensus.cc:2932] T 802538381de843ab8d326014a0b4ec15 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:38.101675 786 ts_tablet_manager.cc:902] T 489db424748b4a73966b047e81f36c83 P f1525cbc82a24d13b65f136695c903af: Initiating tablet copy from peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:38.112001 721 tablet_bootstrap.cc:492] T 5eccb9416d7d438893307f13dd4e73dc P 64998323a1f4482d8704bf20594d13a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:38.112506 721 tablet_bootstrap.cc:492] T 5eccb9416d7d438893307f13dd4e73dc P 64998323a1f4482d8704bf20594d13a5: Bootstrap complete.
I20240501 08:12:38.112810 721 ts_tablet_manager.cc:1367] T 5eccb9416d7d438893307f13dd4e73dc P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.529s user 0.065s sys 0.002s
I20240501 08:12:38.113601 721 raft_consensus.cc:348] T 5eccb9416d7d438893307f13dd4e73dc P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
I20240501 08:12:38.113855 721 raft_consensus.cc:729] T 5eccb9416d7d438893307f13dd4e73dc P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: LEARNER
I20240501 08:12:38.118325 30669 raft_consensus.cc:1203] T 5536362915034f3eb86138b45d71b8ad P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:38.118727 27619 catalog_manager.cc:5484] T 802538381de843ab8d326014a0b4ec15 P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 4 to 5, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:38.125587 30935 raft_consensus.cc:1261] T 00a73d23b12d48a2966f94789db4369a P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:38.126386 27889 raft_consensus.cc:2932] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:38.148018 786 tablet_copy_client.cc:308] T 489db424748b4a73966b047e81f36c83 P f1525cbc82a24d13b65f136695c903af: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.132:41323
I20240501 08:12:38.151672 28172 tablet_copy_service.cc:140] P 91d11eae5ef6432ca86eb8ed446b6ea6: Received BeginTabletCopySession request for tablet 489db424748b4a73966b047e81f36c83 from peer f1525cbc82a24d13b65f136695c903af ({username='slave'} at 127.21.84.130:41275)
I20240501 08:12:38.151862 28172 tablet_copy_service.cc:161] P 91d11eae5ef6432ca86eb8ed446b6ea6: Beginning new tablet copy session on tablet 489db424748b4a73966b047e81f36c83 from peer f1525cbc82a24d13b65f136695c903af at {username='slave'} at 127.21.84.130:41275: session id = f1525cbc82a24d13b65f136695c903af-489db424748b4a73966b047e81f36c83
I20240501 08:12:38.156704 765 tablet_copy_client.cc:308] T d6bb70263134414b9945a87087b55a2c P aad81047009a4ffcbc450b60030583b5: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.130:36503
I20240501 08:12:38.160746 30935 raft_consensus.cc:1203] T b12559fc416a4f3cbffa018611226bcc P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:38.166060 28172 tablet_copy_source_session.cc:205] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:38.167606 786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 489db424748b4a73966b047e81f36c83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:38.171363 27912 tablet_copy_service.cc:140] P f1525cbc82a24d13b65f136695c903af: Received BeginTabletCopySession request for tablet d6bb70263134414b9945a87087b55a2c from peer aad81047009a4ffcbc450b60030583b5 ({username='slave'} at 127.21.84.131:37997)
I20240501 08:12:38.171511 27912 tablet_copy_service.cc:161] P f1525cbc82a24d13b65f136695c903af: Beginning new tablet copy session on tablet d6bb70263134414b9945a87087b55a2c from peer aad81047009a4ffcbc450b60030583b5 at {username='slave'} at 127.21.84.131:37997: session id = aad81047009a4ffcbc450b60030583b5-d6bb70263134414b9945a87087b55a2c
I20240501 08:12:38.174844 786 tablet_copy_client.cc:783] T 489db424748b4a73966b047e81f36c83 P f1525cbc82a24d13b65f136695c903af: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:38.175163 786 tablet_copy_client.cc:647] T 489db424748b4a73966b047e81f36c83 P f1525cbc82a24d13b65f136695c903af: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:38.179404 27912 tablet_copy_source_session.cc:205] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:38.182359 765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6bb70263134414b9945a87087b55a2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:38.186002 786 tablet_copy_client.cc:523] T 489db424748b4a73966b047e81f36c83 P f1525cbc82a24d13b65f136695c903af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:38.189550 786 tablet_bootstrap.cc:492] T 489db424748b4a73966b047e81f36c83 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:38.200187 765 tablet_copy_client.cc:783] T d6bb70263134414b9945a87087b55a2c P aad81047009a4ffcbc450b60030583b5: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:38.201349 765 tablet_copy_client.cc:647] T d6bb70263134414b9945a87087b55a2c P aad81047009a4ffcbc450b60030583b5: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:38.205367 765 tablet_copy_client.cc:523] T d6bb70263134414b9945a87087b55a2c P aad81047009a4ffcbc450b60030583b5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:38.207479 27759 consensus_queue.cc:237] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:12:38.209589 765 tablet_bootstrap.cc:492] T d6bb70263134414b9945a87087b55a2c P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:38.210093 27889 raft_consensus.cc:1261] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:38.210748 312 consensus_queue.cc:1035] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:38.211133 30670 raft_consensus.cc:1261] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:38.211691 312 consensus_queue.cc:1035] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:38.127106 32378 consensus_queue.cc:1035] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:38.216477 312 raft_consensus.cc:2932] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:38.217006 28152 raft_consensus.cc:1261] T 00a73d23b12d48a2966f94789db4369a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:38.217610 792 consensus_queue.cc:1035] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:38.218755 27889 raft_consensus.cc:2932] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:38.219512 30670 raft_consensus.cc:1261] T 00a73d23b12d48a2966f94789db4369a P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:38.220033 792 consensus_queue.cc:1035] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:38.220652 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 802538381de843ab8d326014a0b4ec15 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:38.227114 30670 raft_consensus.cc:2932] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:38.235116 27615 catalog_manager.cc:5484] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af reported cstate change: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:38.239919 28000 tablet_service.cc:1514] Processing DeleteTablet for tablet 802538381de843ab8d326014a0b4ec15 with delete_type TABLET_DATA_TOMBSTONED (TS aad81047009a4ffcbc450b60030583b5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:38.252579 765 tablet_bootstrap.cc:492] T d6bb70263134414b9945a87087b55a2c P aad81047009a4ffcbc450b60030583b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:38.252808 456 raft_consensus.cc:1050] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11: attempting to promote NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 to VOTER
I20240501 08:12:38.253093 765 tablet_bootstrap.cc:492] T d6bb70263134414b9945a87087b55a2c P aad81047009a4ffcbc450b60030583b5: Bootstrap complete.
I20240501 08:12:38.253373 765 ts_tablet_manager.cc:1367] T d6bb70263134414b9945a87087b55a2c P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.044s user 0.017s sys 0.000s
I20240501 08:12:38.254520 456 consensus_queue.cc:237] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:12:38.257269 30671 raft_consensus.cc:1261] T 946add184f4a46f28fd631e8a3bc0ba7 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:38.258284 606 consensus_queue.cc:1035] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:38.259112 765 raft_consensus.cc:348] T d6bb70263134414b9945a87087b55a2c P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:12:38.259351 765 raft_consensus.cc:729] T d6bb70263134414b9945a87087b55a2c P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: LEARNER
I20240501 08:12:38.259547 765 consensus_queue.cc:260] T d6bb70263134414b9945a87087b55a2c P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:12:38.262459 765 ts_tablet_manager.cc:1398] T d6bb70263134414b9945a87087b55a2c P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
W20240501 08:12:38.207605 30740 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.133:38293, user_credentials={real_user=slave}} blocked reactor thread for 74287.4us
I20240501 08:12:38.270190 362 raft_consensus.cc:1050] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47: attempting to promote NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 to VOTER
I20240501 08:12:38.271479 362 consensus_queue.cc:237] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:12:38.277272 721 consensus_queue.cc:260] T 5eccb9416d7d438893307f13dd4e73dc P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
I20240501 08:12:38.278322 721 ts_tablet_manager.cc:1398] T 5eccb9416d7d438893307f13dd4e73dc P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.165s user 0.005s sys 0.000s
I20240501 08:12:38.281558 28018 raft_consensus.cc:1261] T 946add184f4a46f28fd631e8a3bc0ba7 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:38.282213 27912 tablet_copy_service.cc:342] P f1525cbc82a24d13b65f136695c903af: Request end of tablet copy session aad81047009a4ffcbc450b60030583b5-d6bb70263134414b9945a87087b55a2c received from {username='slave'} at 127.21.84.131:37997
I20240501 08:12:38.282313 27912 tablet_copy_service.cc:433] P f1525cbc82a24d13b65f136695c903af: ending tablet copy session aad81047009a4ffcbc450b60030583b5-d6bb70263134414b9945a87087b55a2c on tablet d6bb70263134414b9945a87087b55a2c with peer aad81047009a4ffcbc450b60030583b5
I20240501 08:12:38.282472 456 consensus_queue.cc:1035] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:38.290334 550 raft_consensus.cc:2932] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:38.295451 28152 raft_consensus.cc:2932] T 00a73d23b12d48a2966f94789db4369a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:38.296466 30935 raft_consensus.cc:2932] T 00a73d23b12d48a2966f94789db4369a P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:38.307158 27620 catalog_manager.cc:5484] T 00a73d23b12d48a2966f94789db4369a P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:38.312780 786 tablet_bootstrap.cc:492] T 489db424748b4a73966b047e81f36c83 P f1525cbc82a24d13b65f136695c903af: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:38.313162 786 tablet_bootstrap.cc:492] T 489db424748b4a73966b047e81f36c83 P f1525cbc82a24d13b65f136695c903af: Bootstrap complete.
I20240501 08:12:38.313387 786 ts_tablet_manager.cc:1367] T 489db424748b4a73966b047e81f36c83 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.124s user 0.020s sys 0.050s
I20240501 08:12:38.314016 786 raft_consensus.cc:348] T 489db424748b4a73966b047e81f36c83 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:38.314225 786 raft_consensus.cc:729] T 489db424748b4a73966b047e81f36c83 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: LEARNER
I20240501 08:12:38.314399 786 consensus_queue.cc:260] T 489db424748b4a73966b047e81f36c83 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:38.314610 28042 tablet_copy_service.cc:342] P aad81047009a4ffcbc450b60030583b5: Request end of tablet copy session 64998323a1f4482d8704bf20594d13a5-5eccb9416d7d438893307f13dd4e73dc received from {username='slave'} at 127.21.84.135:55053
I20240501 08:12:38.314721 28042 tablet_copy_service.cc:433] P aad81047009a4ffcbc450b60030583b5: ending tablet copy session 64998323a1f4482d8704bf20594d13a5-5eccb9416d7d438893307f13dd4e73dc on tablet 5eccb9416d7d438893307f13dd4e73dc with peer 64998323a1f4482d8704bf20594d13a5
I20240501 08:12:38.316277 786 ts_tablet_manager.cc:1398] T 489db424748b4a73966b047e81f36c83 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240501 08:12:38.316900 28172 tablet_copy_service.cc:342] P 91d11eae5ef6432ca86eb8ed446b6ea6: Request end of tablet copy session f1525cbc82a24d13b65f136695c903af-489db424748b4a73966b047e81f36c83 received from {username='slave'} at 127.21.84.130:41275
I20240501 08:12:38.317004 28172 tablet_copy_service.cc:433] P 91d11eae5ef6432ca86eb8ed446b6ea6: ending tablet copy session f1525cbc82a24d13b65f136695c903af-489db424748b4a73966b047e81f36c83 on tablet 489db424748b4a73966b047e81f36c83 with peer f1525cbc82a24d13b65f136695c903af
I20240501 08:12:38.331058 27759 raft_consensus.cc:1261] T 946add184f4a46f28fd631e8a3bc0ba7 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:38.334008 30669 raft_consensus.cc:2932] T 00a73d23b12d48a2966f94789db4369a P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:38.337899 606 consensus_queue.cc:1035] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:38.338393 596 raft_consensus.cc:1050] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af: attempting to promote NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 to VOTER
I20240501 08:12:38.339437 596 consensus_queue.cc:237] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:12:38.340514 312 raft_consensus.cc:1050] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43: attempting to promote NON_VOTER f1525cbc82a24d13b65f136695c903af to VOTER
I20240501 08:12:38.341502 312 consensus_queue.cc:237] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:38.345887 30936 raft_consensus.cc:1261] T 683aa9595f2e456c943fc4d16f22e83c P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:38.346797 746 consensus_queue.cc:1035] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:38.348173 27761 raft_consensus.cc:1261] T 683aa9595f2e456c943fc4d16f22e83c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:38.350754 746 consensus_queue.cc:1035] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:38.356957 30937 raft_consensus.cc:1261] T c4851258508d46c88493503a6257f365 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:38.358173 817 consensus_queue.cc:1035] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:38.363617 813 ts_tablet_manager.cc:902] T 6c324b1136ce429290d7286399bb7121 P 91d11eae5ef6432ca86eb8ed446b6ea6: Initiating tablet copy from peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
W20240501 08:12:38.386171 27825 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.134:39933, user_credentials={real_user=slave}} blocked reactor thread for 66551.7us
I20240501 08:12:38.387794 28150 raft_consensus.cc:1261] T c4851258508d46c88493503a6257f365 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:38.389968 812 consensus_queue.cc:1035] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:38.390714 596 raft_consensus.cc:1050] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af: attempting to promote NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 to VOTER
I20240501 08:12:38.391547 596 consensus_queue.cc:237] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:12:38.397655 813 tablet_copy_client.cc:308] T 6c324b1136ce429290d7286399bb7121 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.136:46845
I20240501 08:12:38.430307 28022 consensus_queue.cc:237] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:12:38.434248 30804 raft_consensus.cc:1203] T 5eccb9416d7d438893307f13dd4e73dc P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:38.472311 563 raft_consensus.cc:1050] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43: attempting to promote NON_VOTER 64998323a1f4482d8704bf20594d13a5 to VOTER
I20240501 08:12:38.473402 563 consensus_queue.cc:237] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:12:38.481040 27889 raft_consensus.cc:1203] T 489db424748b4a73966b047e81f36c83 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:38.490275 30803 raft_consensus.cc:1261] T c4851258508d46c88493503a6257f365 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:38.493330 30804 raft_consensus.cc:1261] T 683aa9595f2e456c943fc4d16f22e83c P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:38.495004 550 raft_consensus.cc:1050] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5: attempting to promote NON_VOTER f1525cbc82a24d13b65f136695c903af to VOTER
I20240501 08:12:38.495505 28152 raft_consensus.cc:1261] T 8ab0410a98974002ab68dd0f6a456d59 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:38.495882 550 consensus_queue.cc:237] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:38.496133 563 consensus_queue.cc:1035] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:38.497840 30934 raft_consensus.cc:1261] T 8ab0410a98974002ab68dd0f6a456d59 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:38.498523 30804 raft_consensus.cc:1261] T 8ab0410a98974002ab68dd0f6a456d59 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:38.499171 623 consensus_queue.cc:1035] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:38.499804 623 consensus_queue.cc:1035] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:38.504400 749 tablet_bootstrap.cc:492] T e40fb0a0d49f44a1a36c33412c630077 P 31811708c6774037b82df44024b229b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:38.504403 784 raft_consensus.cc:2932] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:38.504994 749 tablet_bootstrap.cc:492] T e40fb0a0d49f44a1a36c33412c630077 P 31811708c6774037b82df44024b229b6: Bootstrap complete.
I20240501 08:12:38.505328 749 ts_tablet_manager.cc:1367] T e40fb0a0d49f44a1a36c33412c630077 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.666s user 0.055s sys 0.010s
I20240501 08:12:38.506348 749 raft_consensus.cc:348] T e40fb0a0d49f44a1a36c33412c630077 P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
I20240501 08:12:38.506624 749 raft_consensus.cc:729] T e40fb0a0d49f44a1a36c33412c630077 P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: LEARNER
I20240501 08:12:38.512117 30934 raft_consensus.cc:2932] T 8ab0410a98974002ab68dd0f6a456d59 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:38.506879 749 consensus_queue.cc:260] T e40fb0a0d49f44a1a36c33412c630077 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
I20240501 08:12:38.514961 749 ts_tablet_manager.cc:1398] T e40fb0a0d49f44a1a36c33412c630077 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240501 08:12:38.515892 28172 tablet_copy_service.cc:342] P 91d11eae5ef6432ca86eb8ed446b6ea6: Request end of tablet copy session 31811708c6774037b82df44024b229b6-e40fb0a0d49f44a1a36c33412c630077 received from {username='slave'} at 127.21.84.133:45793
I20240501 08:12:38.516027 28172 tablet_copy_service.cc:433] P 91d11eae5ef6432ca86eb8ed446b6ea6: ending tablet copy session 31811708c6774037b82df44024b229b6-e40fb0a0d49f44a1a36c33412c630077 on tablet e40fb0a0d49f44a1a36c33412c630077 with peer 31811708c6774037b82df44024b229b6
I20240501 08:12:38.516726 746 consensus_queue.cc:1035] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:38.517179 27620 catalog_manager.cc:5484] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index 2 to 3, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:38.523008 30804 raft_consensus.cc:2932] T 8ab0410a98974002ab68dd0f6a456d59 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:38.524379 30671 raft_consensus.cc:1261] T 131eac1e19264e00b76500d2ba42697a P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:38.526684 32293 raft_consensus.cc:1050] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6: attempting to promote NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 to VOTER
I20240501 08:12:38.527743 32293 consensus_queue.cc:237] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:12:38.528623 793 consensus_queue.cc:1035] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20240501 08:12:38.533949 27889 raft_consensus.cc:1261] T 131eac1e19264e00b76500d2ba42697a P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:38.543834 550 consensus_queue.cc:1035] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:38.546658 30536 raft_consensus.cc:1261] T 131eac1e19264e00b76500d2ba42697a P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:38.562215 812 consensus_queue.cc:1035] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:38.568259 30804 consensus_queue.cc:237] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:12:38.581547 32641 raft_consensus.cc:1050] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5: attempting to promote NON_VOTER a29cd590999b4f32849ff5689cd53c11 to VOTER
I20240501 08:12:38.583112 32641 consensus_queue.cc:237] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:12:38.600180 30934 raft_consensus.cc:1261] T b12559fc416a4f3cbffa018611226bcc P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:38.600960 798 consensus_queue.cc:1035] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:38.601661 30671 raft_consensus.cc:1261] T 5536362915034f3eb86138b45d71b8ad P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:38.602561 27891 raft_consensus.cc:1261] T 5536362915034f3eb86138b45d71b8ad P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:38.604521 307 consensus_queue.cc:1035] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:38.605129 307 consensus_queue.cc:1035] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:38.611644 792 consensus_queue.cc:1035] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:38.613669 30669 raft_consensus.cc:1261] T 00a73d23b12d48a2966f94789db4369a P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:38.614049 28151 raft_consensus.cc:1261] T 00a73d23b12d48a2966f94789db4369a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:38.614218 27890 raft_consensus.cc:1261] T b12559fc416a4f3cbffa018611226bcc P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:38.614710 791 consensus_queue.cc:1035] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:38.615374 655 consensus_queue.cc:1035] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:38.615576 791 consensus_queue.cc:1035] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:38.616681 804 tablet_replica.cc:330] T 802538381de843ab8d326014a0b4ec15 P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:38.620249 791 raft_consensus.cc:2932] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:38.622205 30669 raft_consensus.cc:2932] T 00a73d23b12d48a2966f94789db4369a P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:38.624644 28151 raft_consensus.cc:2932] T 00a73d23b12d48a2966f94789db4369a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:38.626344 27605 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 00a73d23b12d48a2966f94789db4369a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:38.652053 456 raft_consensus.cc:2932] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:38.653043 30937 raft_consensus.cc:1261] T 4fd595c79cdc43f9a8afadb6b548bf64 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:38.653975 28151 raft_consensus.cc:1261] T 4fd595c79cdc43f9a8afadb6b548bf64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:38.654548 30669 raft_consensus.cc:1261] T 4fd595c79cdc43f9a8afadb6b548bf64 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:38.654534 817 consensus_queue.cc:1035] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:38.655097 817 consensus_queue.cc:1035] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:38.656275 27758 raft_consensus.cc:2932] T 946add184f4a46f28fd631e8a3bc0ba7 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:38.660554 596 consensus_queue.cc:1035] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:38.661926 30669 raft_consensus.cc:2932] T 946add184f4a46f28fd631e8a3bc0ba7 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:38.667685 746 raft_consensus.cc:2932] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:38.674752 27760 raft_consensus.cc:2932] T 683aa9595f2e456c943fc4d16f22e83c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:38.686414 791 raft_consensus.cc:2932] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:38.692759 28149 raft_consensus.cc:2932] T 8ab0410a98974002ab68dd0f6a456d59 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:38.703114 27891 raft_consensus.cc:2932] T 131eac1e19264e00b76500d2ba42697a P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:38.706784 30672 raft_consensus.cc:2932] T 131eac1e19264e00b76500d2ba42697a P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:38.708735 817 raft_consensus.cc:2932] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:38.715140 30802 raft_consensus.cc:2932] T 683aa9595f2e456c943fc4d16f22e83c P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:38.714164 28151 raft_consensus.cc:2932] T 4fd595c79cdc43f9a8afadb6b548bf64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
W20240501 08:12:38.722151 28087 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.131:36305, user_credentials={real_user=slave}} blocked reactor thread for 446296us
I20240501 08:12:38.723274 27620 catalog_manager.cc:5484] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af reported cstate change: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:38.716174 30670 raft_consensus.cc:2932] T 4fd595c79cdc43f9a8afadb6b548bf64 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:38.728927 30957 tablet_copy_service.cc:140] P a29cd590999b4f32849ff5689cd53c11: Received BeginTabletCopySession request for tablet 6c324b1136ce429290d7286399bb7121 from peer 91d11eae5ef6432ca86eb8ed446b6ea6 ({username='slave'} at 127.21.84.132:58509)
I20240501 08:12:38.733614 27890 raft_consensus.cc:1261] T 103e1394bfe34a86ba1d9487d75fd512 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:38.734124 784 consensus_queue.cc:1035] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:38.734306 30537 raft_consensus.cc:1261] T 103e1394bfe34a86ba1d9487d75fd512 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:38.734990 30957 tablet_copy_service.cc:161] P a29cd590999b4f32849ff5689cd53c11: Beginning new tablet copy session on tablet 6c324b1136ce429290d7286399bb7121 from peer 91d11eae5ef6432ca86eb8ed446b6ea6 at {username='slave'} at 127.21.84.132:58509: session id = 91d11eae5ef6432ca86eb8ed446b6ea6-6c324b1136ce429290d7286399bb7121
I20240501 08:12:38.735550 30669 raft_consensus.cc:1261] T 103e1394bfe34a86ba1d9487d75fd512 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:38.737313 623 consensus_queue.cc:1035] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:38.737869 623 consensus_queue.cc:1035] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:38.739485 30957 tablet_copy_source_session.cc:205] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:38.737105 30802 raft_consensus.cc:1261] T 28a7c90dfdf4459aae99c552cfba5b4b P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:12:38.689646 27957 consensus_peers.cc:485] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5 -> Peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): Couldn't send request to peer a29cd590999b4f32849ff5689cd53c11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 28a7c90dfdf4459aae99c552cfba5b4b. This is attempt 1: this message will repeat every 5th retry.
W20240501 08:12:38.746083 27957 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.136:46845, user_credentials={real_user=slave}} blocked reactor thread for 56480.6us
I20240501 08:12:38.732750 28018 raft_consensus.cc:1261] T 5536362915034f3eb86138b45d71b8ad P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:38.741724 32641 consensus_queue.cc:1035] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:38.750006 804 raft_consensus.cc:2229] T 802538381de843ab8d326014a0b4ec15 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:38.750370 804 raft_consensus.cc:2258] T 802538381de843ab8d326014a0b4ec15 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:38.751765 804 ts_tablet_manager.cc:1875] T 802538381de843ab8d326014a0b4ec15 P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:38.759402 623 raft_consensus.cc:2932] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:38.760818 817 raft_consensus.cc:2932] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:38.765822 30537 raft_consensus.cc:2932] T 103e1394bfe34a86ba1d9487d75fd512 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:38.770136 30935 raft_consensus.cc:2932] T c4851258508d46c88493503a6257f365 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:38.773725 27890 raft_consensus.cc:2932] T 103e1394bfe34a86ba1d9487d75fd512 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:38.778657 27616 catalog_manager.cc:5484] T 103e1394bfe34a86ba1d9487d75fd512 P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:38.778987 28152 raft_consensus.cc:2932] T c4851258508d46c88493503a6257f365 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:38.784613 30669 raft_consensus.cc:2932] T 103e1394bfe34a86ba1d9487d75fd512 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:38.793591 30537 raft_consensus.cc:1261] T 28a7c90dfdf4459aae99c552cfba5b4b P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:38.802263 30935 raft_consensus.cc:2932] T b12559fc416a4f3cbffa018611226bcc P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:38.665377 27612 catalog_manager.cc:5484] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index 2 to 3, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:38.784214 655 raft_consensus.cc:2932] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:38.790757 28020 raft_consensus.cc:2932] T 946add184f4a46f28fd631e8a3bc0ba7 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:38.794126 798 consensus_queue.cc:1035] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:38.745703 28022 raft_consensus.cc:1203] T d6bb70263134414b9945a87087b55a2c P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:38.832129 27890 raft_consensus.cc:2932] T b12559fc416a4f3cbffa018611226bcc P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:38.845060 32641 raft_consensus.cc:2932] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:38.847080 30534 raft_consensus.cc:2932] T 28a7c90dfdf4459aae99c552cfba5b4b P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
W20240501 08:12:38.851575 30607 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.129:43471, user_credentials={real_user=slave}} blocked reactor thread for 173989us
I20240501 08:12:38.855391 30535 raft_consensus.cc:1203] T e40fb0a0d49f44a1a36c33412c630077 P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:38.857271 30934 raft_consensus.cc:2932] T 683aa9595f2e456c943fc4d16f22e83c P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:38.863715 30802 raft_consensus.cc:2932] T 28a7c90dfdf4459aae99c552cfba5b4b P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:38.867384 840 raft_consensus.cc:2932] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:38.867859 30537 raft_consensus.cc:2932] T 131eac1e19264e00b76500d2ba42697a P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:38.868686 27890 consensus_queue.cc:237] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:12:38.871461 28150 raft_consensus.cc:1261] T 4fd595c79cdc43f9a8afadb6b548bf64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:38.872231 32079 consensus_queue.cc:1035] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:38.873060 30671 raft_consensus.cc:1261] T 4fd595c79cdc43f9a8afadb6b548bf64 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:38.875228 32079 consensus_queue.cc:1035] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:38.876235 857 raft_consensus.cc:1050] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5: attempting to promote NON_VOTER 64998323a1f4482d8704bf20594d13a5 to VOTER
I20240501 08:12:38.876946 857 consensus_queue.cc:237] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:12:38.877151 32079 raft_consensus.cc:2932] T 4fd595c79cdc43f9a8afadb6b548bf64 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:38.878199 27890 raft_consensus.cc:2932] T 5536362915034f3eb86138b45d71b8ad P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:38.880782 27607 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4fd595c79cdc43f9a8afadb6b548bf64 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:38.880878 28150 raft_consensus.cc:2932] T 4fd595c79cdc43f9a8afadb6b548bf64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:38.885100 30671 raft_consensus.cc:2932] T 4fd595c79cdc43f9a8afadb6b548bf64 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:38.887480 30804 raft_consensus.cc:2932] T c4851258508d46c88493503a6257f365 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:38.902451 27615 catalog_manager.cc:5484] T 00a73d23b12d48a2966f94789db4369a P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:38.903121 27615 catalog_manager.cc:5484] T 131eac1e19264e00b76500d2ba42697a P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:38.903709 27615 catalog_manager.cc:5484] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:38.903662 30672 raft_consensus.cc:2932] T 5536362915034f3eb86138b45d71b8ad P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
W20240501 08:12:38.904768 847 log.cc:924] Time spent T 4fd595c79cdc43f9a8afadb6b548bf64 P a29cd590999b4f32849ff5689cd53c11: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20240501 08:12:38.909008 30937 consensus_queue.cc:237] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:12:38.911633 27759 raft_consensus.cc:1261] T 946add184f4a46f28fd631e8a3bc0ba7 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:38.912562 606 consensus_queue.cc:1035] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:38.917853 606 raft_consensus.cc:2932] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:38.918720 30804 raft_consensus.cc:1261] T 5eccb9416d7d438893307f13dd4e73dc P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:38.919277 32641 consensus_queue.cc:1035] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:38.920137 28150 raft_consensus.cc:1261] T b12559fc416a4f3cbffa018611226bcc P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:38.922250 27759 raft_consensus.cc:1261] T 5eccb9416d7d438893307f13dd4e73dc P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:38.922072 27758 raft_consensus.cc:2932] T 946add184f4a46f28fd631e8a3bc0ba7 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:38.922752 857 consensus_queue.cc:1035] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:38.927227 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 946add184f4a46f28fd631e8a3bc0ba7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:38.930317 30672 raft_consensus.cc:1261] T 946add184f4a46f28fd631e8a3bc0ba7 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:38.931145 857 consensus_queue.cc:1035] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:38.932147 456 consensus_queue.cc:1035] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:38.934005 813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c324b1136ce429290d7286399bb7121. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:38.934276 27611 catalog_manager.cc:5484] T 28a7c90dfdf4459aae99c552cfba5b4b P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index -1 to 2, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:38.934839 840 consensus_queue.cc:1035] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:38.942252 813 tablet_copy_client.cc:783] T 6c324b1136ce429290d7286399bb7121 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:38.942543 813 tablet_copy_client.cc:647] T 6c324b1136ce429290d7286399bb7121 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:38.943398 30916 tablet_service.cc:1514] Processing DeleteTablet for tablet 00a73d23b12d48a2966f94789db4369a with delete_type TABLET_DATA_TOMBSTONED (TS a29cd590999b4f32849ff5689cd53c11 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54036
I20240501 08:12:38.946020 27616 catalog_manager.cc:5484] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:38.952584 28152 raft_consensus.cc:1261] T 5eccb9416d7d438893307f13dd4e73dc P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:38.953577 27761 raft_consensus.cc:2932] T 5eccb9416d7d438893307f13dd4e73dc P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:38.956342 30672 raft_consensus.cc:2932] T 946add184f4a46f28fd631e8a3bc0ba7 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:38.968401 27615 catalog_manager.cc:5484] T 5536362915034f3eb86138b45d71b8ad P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 4 to 5, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:38.969012 27615 catalog_manager.cc:5484] T 4fd595c79cdc43f9a8afadb6b548bf64 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:38.970974 28022 consensus_queue.cc:237] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:12:38.975728 28150 raft_consensus.cc:2932] T b12559fc416a4f3cbffa018611226bcc P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:38.976625 868 tablet_replica.cc:330] T 00a73d23b12d48a2966f94789db4369a P a29cd590999b4f32849ff5689cd53c11: stopping tablet replica
I20240501 08:12:38.978173 868 raft_consensus.cc:2229] T 00a73d23b12d48a2966f94789db4369a P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:38.978578 868 raft_consensus.cc:2258] T 00a73d23b12d48a2966f94789db4369a P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:38.979230 30916 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fd595c79cdc43f9a8afadb6b548bf64 with delete_type TABLET_DATA_TOMBSTONED (TS a29cd590999b4f32849ff5689cd53c11 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54036
I20240501 08:12:38.978376 27616 catalog_manager.cc:5484] T c4851258508d46c88493503a6257f365 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:38.983374 868 ts_tablet_manager.cc:1875] T 00a73d23b12d48a2966f94789db4369a P a29cd590999b4f32849ff5689cd53c11: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:38.987650 27611 catalog_manager.cc:5484] T 946add184f4a46f28fd631e8a3bc0ba7 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index 3 to 4, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:38.988039 27620 catalog_manager.cc:5031] ChangeConfig:REMOVE_PEER RPC for tablet 4fd595c79cdc43f9a8afadb6b548bf64 with cas_config_opid_index 3: aborting the task: latest config opid_index 4; task opid_index 3
I20240501 08:12:38.988701 27611 catalog_manager.cc:5484] T 5eccb9416d7d438893307f13dd4e73dc P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:38.990492 30536 consensus_queue.cc:237] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:12:38.991230 28152 consensus_queue.cc:237] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:12:38.992995 27890 consensus_queue.cc:237] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:12:38.995013 30671 raft_consensus.cc:1261] T 5536362915034f3eb86138b45d71b8ad P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:38.996379 30803 raft_consensus.cc:1261] T c4851258508d46c88493503a6257f365 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:38.996259 32293 consensus_queue.cc:1035] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:38.996897 32079 consensus_queue.cc:1035] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:38.999814 32079 raft_consensus.cc:2932] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:39.001106 27890 raft_consensus.cc:1261] T 5536362915034f3eb86138b45d71b8ad P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:39.001658 32293 consensus_queue.cc:1035] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.003913 30671 consensus_queue.cc:237] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:12:39.004269 32293 raft_consensus.cc:2932] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:39.006079 307 raft_consensus.cc:1050] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6: attempting to promote NON_VOTER f1525cbc82a24d13b65f136695c903af to VOTER
I20240501 08:12:39.006922 307 consensus_queue.cc:237] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:39.007193 30937 raft_consensus.cc:1261] T 683aa9595f2e456c943fc4d16f22e83c P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:39.008157 362 consensus_queue.cc:1035] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:39.008327 27607 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c4851258508d46c88493503a6257f365 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:39.008375 868 ts_tablet_manager.cc:1888] T 00a73d23b12d48a2966f94789db4369a P a29cd590999b4f32849ff5689cd53c11: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:39.008519 868 log.cc:1196] T 00a73d23b12d48a2966f94789db4369a P a29cd590999b4f32849ff5689cd53c11: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/wals/00a73d23b12d48a2966f94789db4369a
I20240501 08:12:39.011325 868 tablet_replica.cc:330] T 4fd595c79cdc43f9a8afadb6b548bf64 P a29cd590999b4f32849ff5689cd53c11: stopping tablet replica
I20240501 08:12:39.011551 868 raft_consensus.cc:2229] T 4fd595c79cdc43f9a8afadb6b548bf64 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:39.011668 27606 catalog_manager.cc:4830] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): tablet 00a73d23b12d48a2966f94789db4369a (table rebalancer_extra_table_34 [id=1d72487b1d1a4310bbfd3ad3afddbee9]) successfully deleted
I20240501 08:12:39.011796 868 pending_rounds.cc:62] T 4fd595c79cdc43f9a8afadb6b548bf64 P a29cd590999b4f32849ff5689cd53c11: Trying to abort 1 pending ops.
I20240501 08:12:39.011940 868 pending_rounds.cc:68] T 4fd595c79cdc43f9a8afadb6b548bf64 P a29cd590999b4f32849ff5689cd53c11: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7022801544769683456 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "4fd595c79cdc43f9a8afadb6b548bf64" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } } }
I20240501 08:12:39.012413 28152 raft_consensus.cc:1261] T c4851258508d46c88493503a6257f365 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:39.012447 868 raft_consensus.cc:2894] T 4fd595c79cdc43f9a8afadb6b548bf64 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240501 08:12:39.012581 868 raft_consensus.cc:2866] T 4fd595c79cdc43f9a8afadb6b548bf64 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 08:12:39.012687 30669 raft_consensus.cc:1261] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:39.012746 868 raft_consensus.cc:2258] T 4fd595c79cdc43f9a8afadb6b548bf64 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:39.012897 27759 consensus_queue.cc:237] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:39.013904 775 consensus_queue.cc:1035] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:39.014796 811 consensus_queue.cc:1035] T c4851258508d46c88493503a6257f365 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:39.014977 27891 raft_consensus.cc:1261] T 103e1394bfe34a86ba1d9487d75fd512 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:39.015502 312 consensus_queue.cc:1035] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:39.016407 27891 raft_consensus.cc:2932] T 5536362915034f3eb86138b45d71b8ad P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:39.018158 850 raft_consensus.cc:2932] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:39.019148 30937 raft_consensus.cc:2932] T 683aa9595f2e456c943fc4d16f22e83c P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:39.018644 30670 raft_consensus.cc:1261] T 103e1394bfe34a86ba1d9487d75fd512 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:39.022238 312 consensus_queue.cc:1035] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:39.021792 28152 raft_consensus.cc:2932] T c4851258508d46c88493503a6257f365 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:39.023478 30803 raft_consensus.cc:2932] T c4851258508d46c88493503a6257f365 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:39.024266 27761 raft_consensus.cc:1261] T 683aa9595f2e456c943fc4d16f22e83c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:39.027987 312 raft_consensus.cc:2932] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:39.028957 746 consensus_queue.cc:1035] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:39.030321 27890 raft_consensus.cc:2932] T 103e1394bfe34a86ba1d9487d75fd512 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:39.033385 30804 consensus_queue.cc:237] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:39.033586 868 ts_tablet_manager.cc:1875] T 4fd595c79cdc43f9a8afadb6b548bf64 P a29cd590999b4f32849ff5689cd53c11: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:39.034296 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 103e1394bfe34a86ba1d9487d75fd512 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:39.035715 27604 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 683aa9595f2e456c943fc4d16f22e83c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:39.035837 27889 raft_consensus.cc:1261] T 131eac1e19264e00b76500d2ba42697a P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:39.035888 30670 raft_consensus.cc:1261] T 131eac1e19264e00b76500d2ba42697a P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:39.036440 793 consensus_queue.cc:1035] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:39.036970 550 consensus_queue.cc:1035] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:39.036795 30671 raft_consensus.cc:2932] T 5536362915034f3eb86138b45d71b8ad P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:39.036818 27759 raft_consensus.cc:2932] T 683aa9595f2e456c943fc4d16f22e83c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:39.039796 550 raft_consensus.cc:2932] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:39.041862 27889 raft_consensus.cc:1261] T 489db424748b4a73966b047e81f36c83 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:39.042625 584 consensus_queue.cc:1035] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:39.043498 30670 raft_consensus.cc:2932] T 131eac1e19264e00b76500d2ba42697a P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:38.952759 32641 raft_consensus.cc:2932] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:39.045078 27889 raft_consensus.cc:2932] T 131eac1e19264e00b76500d2ba42697a P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:39.047567 28000 tablet_service.cc:1514] Processing DeleteTablet for tablet 946add184f4a46f28fd631e8a3bc0ba7 with delete_type TABLET_DATA_TOMBSTONED (TS aad81047009a4ffcbc450b60030583b5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:39.052414 27605 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 131eac1e19264e00b76500d2ba42697a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:39.055883 27611 catalog_manager.cc:5484] T 103e1394bfe34a86ba1d9487d75fd512 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 08:12:39.068200 30871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.135:33503, user_credentials={real_user=slave}} blocked reactor thread for 153747us
I20240501 08:12:39.069959 27604 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5536362915034f3eb86138b45d71b8ad with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:39.070385 27760 raft_consensus.cc:1261] T 489db424748b4a73966b047e81f36c83 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:39.071194 808 consensus_queue.cc:1035] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:39.073868 30937 raft_consensus.cc:1261] T 1448b8f2d5844ad7aec9bcd3874c5c01 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:39.076311 813 tablet_copy_client.cc:523] T 6c324b1136ce429290d7286399bb7121 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:39.077003 27616 catalog_manager.cc:5484] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 3 to 4, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New cstate: current_term: 1 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:39.077733 27616 catalog_manager.cc:5484] T 5536362915034f3eb86138b45d71b8ad P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:39.078222 27616 catalog_manager.cc:5484] T 131eac1e19264e00b76500d2ba42697a P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:39.080070 813 tablet_bootstrap.cc:492] T 6c324b1136ce429290d7286399bb7121 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:39.083654 30514 tablet_service.cc:1514] Processing DeleteTablet for tablet 103e1394bfe34a86ba1d9487d75fd512 with delete_type TABLET_DATA_TOMBSTONED (TS 31811708c6774037b82df44024b229b6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45078
I20240501 08:12:39.094285 30937 raft_consensus.cc:1261] T b12559fc416a4f3cbffa018611226bcc P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:39.097513 879 consensus_queue.cc:1035] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.097644 775 consensus_queue.cc:1035] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:12:39.099041 30472 consensus_peers.cc:485] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 -> Peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Couldn't send request to peer aad81047009a4ffcbc450b60030583b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1448b8f2d5844ad7aec9bcd3874c5c01. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:39.099720 30536 raft_consensus.cc:1261] T 489db424748b4a73966b047e81f36c83 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:39.100332 32293 consensus_queue.cc:1035] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:39.100061 877 raft_consensus.cc:2932] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:39.100610 30937 raft_consensus.cc:2932] T b12559fc416a4f3cbffa018611226bcc P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:39.105772 27612 catalog_manager.cc:5031] ChangeConfig:REMOVE_PEER RPC for tablet 946add184f4a46f28fd631e8a3bc0ba7 with cas_config_opid_index 3: aborting the task: latest config opid_index 4; task opid_index 3
I20240501 08:12:39.107148 881 tablet_replica.cc:330] T 103e1394bfe34a86ba1d9487d75fd512 P 31811708c6774037b82df44024b229b6: stopping tablet replica
I20240501 08:12:39.110414 881 raft_consensus.cc:2229] T 103e1394bfe34a86ba1d9487d75fd512 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:39.111590 881 raft_consensus.cc:2258] T 103e1394bfe34a86ba1d9487d75fd512 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:39.111752 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b12559fc416a4f3cbffa018611226bcc with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:39.112998 881 ts_tablet_manager.cc:1875] T 103e1394bfe34a86ba1d9487d75fd512 P 31811708c6774037b82df44024b229b6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:39.113133 27620 catalog_manager.cc:5484] T c4851258508d46c88493503a6257f365 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:39.115705 813 tablet_bootstrap.cc:492] T 6c324b1136ce429290d7286399bb7121 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:39.116210 813 tablet_bootstrap.cc:492] T 6c324b1136ce429290d7286399bb7121 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap complete.
I20240501 08:12:39.116562 813 ts_tablet_manager.cc:1367] T 6c324b1136ce429290d7286399bb7121 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.037s user 0.009s sys 0.001s
I20240501 08:12:39.117288 813 raft_consensus.cc:348] T 6c324b1136ce429290d7286399bb7121 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:12:39.117710 813 raft_consensus.cc:729] T 6c324b1136ce429290d7286399bb7121 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: LEARNER
I20240501 08:12:39.007953 861 consensus_queue.cc:1035] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.111s
I20240501 08:12:39.118310 30916 tablet_service.cc:1514] Processing DeleteTablet for tablet c4851258508d46c88493503a6257f365 with delete_type TABLET_DATA_TOMBSTONED (TS a29cd590999b4f32849ff5689cd53c11 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54036
I20240501 08:12:39.121074 30803 raft_consensus.cc:2932] T 5eccb9416d7d438893307f13dd4e73dc P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:39.124164 813 consensus_queue.cc:260] T 6c324b1136ce429290d7286399bb7121 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:12:39.125747 813 ts_tablet_manager.cc:1398] T 6c324b1136ce429290d7286399bb7121 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
I20240501 08:12:39.126894 30784 tablet_service.cc:1514] Processing DeleteTablet for tablet 683aa9595f2e456c943fc4d16f22e83c with delete_type TABLET_DATA_TOMBSTONED (TS 64998323a1f4482d8704bf20594d13a5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38952
I20240501 08:12:39.127099 30514 tablet_service.cc:1514] Processing DeleteTablet for tablet 131eac1e19264e00b76500d2ba42697a with delete_type TABLET_DATA_TOMBSTONED (TS 31811708c6774037b82df44024b229b6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45078
I20240501 08:12:39.133996 27620 catalog_manager.cc:5484] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 5 to 6, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:39.141497 28000 tablet_service.cc:1514] Processing DeleteTablet for tablet 5536362915034f3eb86138b45d71b8ad with delete_type TABLET_DATA_TOMBSTONED (TS aad81047009a4ffcbc450b60030583b5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:39.143898 28131 tablet_service.cc:1514] Processing DeleteTablet for tablet b12559fc416a4f3cbffa018611226bcc with delete_type TABLET_DATA_TOMBSTONED (TS 91d11eae5ef6432ca86eb8ed446b6ea6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:39.145390 30957 tablet_copy_service.cc:342] P a29cd590999b4f32849ff5689cd53c11: Request end of tablet copy session 91d11eae5ef6432ca86eb8ed446b6ea6-6c324b1136ce429290d7286399bb7121 received from {username='slave'} at 127.21.84.132:58509
I20240501 08:12:39.145568 30957 tablet_copy_service.cc:433] P a29cd590999b4f32849ff5689cd53c11: ending tablet copy session 91d11eae5ef6432ca86eb8ed446b6ea6-6c324b1136ce429290d7286399bb7121 on tablet 6c324b1136ce429290d7286399bb7121 with peer 91d11eae5ef6432ca86eb8ed446b6ea6
I20240501 08:12:39.167193 881 ts_tablet_manager.cc:1888] T 103e1394bfe34a86ba1d9487d75fd512 P 31811708c6774037b82df44024b229b6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:39.168061 28150 raft_consensus.cc:2932] T 5eccb9416d7d438893307f13dd4e73dc P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:39.169427 30672 raft_consensus.cc:2932] T 103e1394bfe34a86ba1d9487d75fd512 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:39.174405 884 tablet_replica.cc:330] T 683aa9595f2e456c943fc4d16f22e83c P 64998323a1f4482d8704bf20594d13a5: stopping tablet replica
I20240501 08:12:39.176174 804 ts_tablet_manager.cc:1888] T 802538381de843ab8d326014a0b4ec15 P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:39.176424 804 log.cc:1196] T 802538381de843ab8d326014a0b4ec15 P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/802538381de843ab8d326014a0b4ec15
I20240501 08:12:39.176841 884 raft_consensus.cc:2229] T 683aa9595f2e456c943fc4d16f22e83c P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:39.178030 884 raft_consensus.cc:2258] T 683aa9595f2e456c943fc4d16f22e83c P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:39.178478 887 ts_tablet_manager.cc:902] T 28a7c90dfdf4459aae99c552cfba5b4b P a29cd590999b4f32849ff5689cd53c11: Initiating tablet copy from peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:39.181032 30935 raft_consensus.cc:2932] T 1448b8f2d5844ad7aec9bcd3874c5c01 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:39.188958 884 ts_tablet_manager.cc:1875] T 683aa9595f2e456c943fc4d16f22e83c P 64998323a1f4482d8704bf20594d13a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:39.189952 808 raft_consensus.cc:2932] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:39.179607 523 raft_consensus.cc:2932] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:39.191754 885 tablet_replica.cc:330] T b12559fc416a4f3cbffa018611226bcc P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:39.191948 885 raft_consensus.cc:2229] T b12559fc416a4f3cbffa018611226bcc P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:39.192158 885 raft_consensus.cc:2258] T b12559fc416a4f3cbffa018611226bcc P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:39.192843 804 tablet_replica.cc:330] T 946add184f4a46f28fd631e8a3bc0ba7 P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:39.193032 804 raft_consensus.cc:2229] T 946add184f4a46f28fd631e8a3bc0ba7 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:39.193140 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet 802538381de843ab8d326014a0b4ec15 (table rebalance_test_table_9 [id=e6efea2767a741beae5c93339e7712ba]) successfully deleted
I20240501 08:12:39.193226 804 raft_consensus.cc:2258] T 946add184f4a46f28fd631e8a3bc0ba7 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:39.194147 804 ts_tablet_manager.cc:1875] T 946add184f4a46f28fd631e8a3bc0ba7 P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:39.197625 27892 raft_consensus.cc:2932] T 489db424748b4a73966b047e81f36c83 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:39.198119 30671 raft_consensus.cc:2932] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:39.198464 30536 raft_consensus.cc:2932] T 489db424748b4a73966b047e81f36c83 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:39.205801 881 log.cc:1196] T 103e1394bfe34a86ba1d9487d75fd512 P 31811708c6774037b82df44024b229b6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/wals/103e1394bfe34a86ba1d9487d75fd512
I20240501 08:12:39.206112 27616 catalog_manager.cc:5484] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:39.206867 27607 catalog_manager.cc:4830] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): tablet 103e1394bfe34a86ba1d9487d75fd512 (table rebalancer_extra_table_10 [id=6377dbf65f184f9cbe44f12bd6a0405f]) successfully deleted
I20240501 08:12:39.208175 881 tablet_replica.cc:330] T 131eac1e19264e00b76500d2ba42697a P 31811708c6774037b82df44024b229b6: stopping tablet replica
I20240501 08:12:39.208426 881 raft_consensus.cc:2229] T 131eac1e19264e00b76500d2ba42697a P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:39.208688 881 raft_consensus.cc:2258] T 131eac1e19264e00b76500d2ba42697a P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:39.208495 28020 raft_consensus.cc:2932] T 5536362915034f3eb86138b45d71b8ad P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:39.210312 27619 catalog_manager.cc:5484] T 489db424748b4a73966b047e81f36c83 P f1525cbc82a24d13b65f136695c903af reported cstate change: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:39.211817 881 ts_tablet_manager.cc:1875] T 131eac1e19264e00b76500d2ba42697a P 31811708c6774037b82df44024b229b6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:39.213637 868 ts_tablet_manager.cc:1888] T 4fd595c79cdc43f9a8afadb6b548bf64 P a29cd590999b4f32849ff5689cd53c11: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:39.213783 868 log.cc:1196] T 4fd595c79cdc43f9a8afadb6b548bf64 P a29cd590999b4f32849ff5689cd53c11: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/wals/4fd595c79cdc43f9a8afadb6b548bf64
I20240501 08:12:39.214484 868 tablet_replica.cc:330] T c4851258508d46c88493503a6257f365 P a29cd590999b4f32849ff5689cd53c11: stopping tablet replica
I20240501 08:12:39.214712 868 raft_consensus.cc:2229] T c4851258508d46c88493503a6257f365 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:39.214782 27606 catalog_manager.cc:4830] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): tablet 4fd595c79cdc43f9a8afadb6b548bf64 (table rebalancer_extra_table_30 [id=9da53f67716d4d2892b2ff2b43922ec5]) successfully deleted
I20240501 08:12:39.215092 868 raft_consensus.cc:2258] T c4851258508d46c88493503a6257f365 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:39.215979 868 ts_tablet_manager.cc:1875] T c4851258508d46c88493503a6257f365 P a29cd590999b4f32849ff5689cd53c11: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:39.216336 27760 raft_consensus.cc:2932] T 489db424748b4a73966b047e81f36c83 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:39.221892 881 ts_tablet_manager.cc:1888] T 131eac1e19264e00b76500d2ba42697a P 31811708c6774037b82df44024b229b6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:39.224017 881 log.cc:1196] T 131eac1e19264e00b76500d2ba42697a P 31811708c6774037b82df44024b229b6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/wals/131eac1e19264e00b76500d2ba42697a
I20240501 08:12:39.224673 885 ts_tablet_manager.cc:1875] T b12559fc416a4f3cbffa018611226bcc P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:39.225011 27607 catalog_manager.cc:4830] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): tablet 131eac1e19264e00b76500d2ba42697a (table rebalancer_extra_table_6 [id=2b1f439c36b24b19bec52d25077c330d]) successfully deleted
W20240501 08:12:39.227152 27958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.133:38293, user_credentials={real_user=slave}} blocked reactor thread for 166684us
I20240501 08:12:39.228365 27890 raft_consensus.cc:1261] T b12559fc416a4f3cbffa018611226bcc P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:39.237028 28150 consensus_queue.cc:237] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:39.240222 27892 raft_consensus.cc:1261] T 489db424748b4a73966b047e81f36c83 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:39.240752 808 consensus_queue.cc:1035] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:39.245889 30536 raft_consensus.cc:1261] T 489db424748b4a73966b047e81f36c83 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:39.246748 808 consensus_queue.cc:1035] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:39.247210 808 raft_consensus.cc:2932] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:39.251822 27892 raft_consensus.cc:2932] T 489db424748b4a73966b047e81f36c83 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:39.258401 884 ts_tablet_manager.cc:1888] T 683aa9595f2e456c943fc4d16f22e83c P 64998323a1f4482d8704bf20594d13a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:39.257551 30536 raft_consensus.cc:2932] T 489db424748b4a73966b047e81f36c83 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:39.267477 884 log.cc:1196] T 683aa9595f2e456c943fc4d16f22e83c P 64998323a1f4482d8704bf20594d13a5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal/wals/683aa9595f2e456c943fc4d16f22e83c
I20240501 08:12:39.269738 655 consensus_queue.cc:1035] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.272972 817 raft_consensus.cc:1050] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af: attempting to promote NON_VOTER aad81047009a4ffcbc450b60030583b5 to VOTER
I20240501 08:12:39.273803 817 consensus_queue.cc:237] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } }
I20240501 08:12:39.274047 27890 raft_consensus.cc:2932] T b12559fc416a4f3cbffa018611226bcc P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:39.276057 27605 catalog_manager.cc:4830] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): tablet 683aa9595f2e456c943fc4d16f22e83c (table rebalancer_extra_table_2 [id=6fb79b6584164a1c9205937b7afef405]) successfully deleted
I20240501 08:12:39.276571 804 ts_tablet_manager.cc:1888] T 946add184f4a46f28fd631e8a3bc0ba7 P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:39.278038 27616 catalog_manager.cc:5484] T 489db424748b4a73966b047e81f36c83 P f1525cbc82a24d13b65f136695c903af reported cstate change: config changed from index 3 to 4, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:39.281426 804 log.cc:1196] T 946add184f4a46f28fd631e8a3bc0ba7 P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/946add184f4a46f28fd631e8a3bc0ba7
I20240501 08:12:39.282265 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet 946add184f4a46f28fd631e8a3bc0ba7 (table rebalancer_extra_table_62 [id=0f8f4d8350da4a678e64ef5f70c8df92]) successfully deleted
I20240501 08:12:39.282740 804 tablet_replica.cc:330] T 5536362915034f3eb86138b45d71b8ad P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:39.282969 804 raft_consensus.cc:2229] T 5536362915034f3eb86138b45d71b8ad P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:39.283785 804 raft_consensus.cc:2258] T 5536362915034f3eb86138b45d71b8ad P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:39.284737 27741 tablet_service.cc:1514] Processing DeleteTablet for tablet 489db424748b4a73966b047e81f36c83 with delete_type TABLET_DATA_TOMBSTONED (TS 6331f77dae9d452184e8cd0cbf5f1b43 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:39.285434 896 tablet_replica.cc:330] T 489db424748b4a73966b047e81f36c83 P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:39.285673 896 raft_consensus.cc:2229] T 489db424748b4a73966b047e81f36c83 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:39.285897 896 raft_consensus.cc:2258] T 489db424748b4a73966b047e81f36c83 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:39.288545 868 ts_tablet_manager.cc:1888] T c4851258508d46c88493503a6257f365 P a29cd590999b4f32849ff5689cd53c11: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:39.288937 804 ts_tablet_manager.cc:1875] T 5536362915034f3eb86138b45d71b8ad P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:39.289187 868 log.cc:1196] T c4851258508d46c88493503a6257f365 P a29cd590999b4f32849ff5689cd53c11: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/wals/c4851258508d46c88493503a6257f365
I20240501 08:12:39.291294 27606 catalog_manager.cc:4830] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): tablet c4851258508d46c88493503a6257f365 (table rebalancer_extra_table_14 [id=bd2ed215ec044ce782fe6f7c802797e6]) successfully deleted
I20240501 08:12:39.291929 887 tablet_copy_client.cc:308] T 28a7c90dfdf4459aae99c552cfba5b4b P a29cd590999b4f32849ff5689cd53c11: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.131:36305
I20240501 08:12:39.292520 28042 tablet_copy_service.cc:140] P aad81047009a4ffcbc450b60030583b5: Received BeginTabletCopySession request for tablet 28a7c90dfdf4459aae99c552cfba5b4b from peer a29cd590999b4f32849ff5689cd53c11 ({username='slave'} at 127.21.84.136:51325)
I20240501 08:12:39.292649 28042 tablet_copy_service.cc:161] P aad81047009a4ffcbc450b60030583b5: Beginning new tablet copy session on tablet 28a7c90dfdf4459aae99c552cfba5b4b from peer a29cd590999b4f32849ff5689cd53c11 at {username='slave'} at 127.21.84.136:51325: session id = a29cd590999b4f32849ff5689cd53c11-28a7c90dfdf4459aae99c552cfba5b4b
I20240501 08:12:39.294755 896 ts_tablet_manager.cc:1875] T 489db424748b4a73966b047e81f36c83 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:39.295481 28042 tablet_copy_source_session.cc:205] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:39.296880 887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28a7c90dfdf4459aae99c552cfba5b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:39.301761 804 ts_tablet_manager.cc:1888] T 5536362915034f3eb86138b45d71b8ad P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:39.307499 28019 raft_consensus.cc:1261] T d6bb70263134414b9945a87087b55a2c P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:39.309123 817 consensus_queue.cc:1035] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:39.312317 896 ts_tablet_manager.cc:1888] T 489db424748b4a73966b047e81f36c83 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:39.312939 896 log.cc:1196] T 489db424748b4a73966b047e81f36c83 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/489db424748b4a73966b047e81f36c83
I20240501 08:12:39.313805 887 tablet_copy_client.cc:783] T 28a7c90dfdf4459aae99c552cfba5b4b P a29cd590999b4f32849ff5689cd53c11: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:39.313994 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet 489db424748b4a73966b047e81f36c83 (table rebalancer_extra_table_8 [id=f51464bcf46d406a807b68e0429a75af]) successfully deleted
I20240501 08:12:39.314146 887 tablet_copy_client.cc:647] T 28a7c90dfdf4459aae99c552cfba5b4b P a29cd590999b4f32849ff5689cd53c11: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:39.316673 887 tablet_copy_client.cc:523] T 28a7c90dfdf4459aae99c552cfba5b4b P a29cd590999b4f32849ff5689cd53c11: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:39.317168 804 log.cc:1196] T 5536362915034f3eb86138b45d71b8ad P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/5536362915034f3eb86138b45d71b8ad
I20240501 08:12:39.318166 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet 5536362915034f3eb86138b45d71b8ad (table rebalance_test_table_15 [id=d3eb6ca7d8cf4bfdb114edb23abb2ed2]) successfully deleted
I20240501 08:12:39.319617 30536 raft_consensus.cc:1261] T d6bb70263134414b9945a87087b55a2c P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:39.320679 455 consensus_queue.cc:1035] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:39.320997 30672 raft_consensus.cc:1261] T d6bb70263134414b9945a87087b55a2c P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:39.321633 455 consensus_queue.cc:1035] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:39.321805 887 tablet_bootstrap.cc:492] T 28a7c90dfdf4459aae99c552cfba5b4b P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:39.328188 817 raft_consensus.cc:2932] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:39.328902 28149 raft_consensus.cc:1203] T 6c324b1136ce429290d7286399bb7121 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:39.328991 30537 raft_consensus.cc:2932] T d6bb70263134414b9945a87087b55a2c P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:39.333271 28019 raft_consensus.cc:2932] T d6bb70263134414b9945a87087b55a2c P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:39.335597 27612 catalog_manager.cc:5484] T d6bb70263134414b9945a87087b55a2c P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:39.341953 887 tablet_bootstrap.cc:492] T 28a7c90dfdf4459aae99c552cfba5b4b P a29cd590999b4f32849ff5689cd53c11: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:39.342466 887 tablet_bootstrap.cc:492] T 28a7c90dfdf4459aae99c552cfba5b4b P a29cd590999b4f32849ff5689cd53c11: Bootstrap complete.
I20240501 08:12:39.342779 887 ts_tablet_manager.cc:1367] T 28a7c90dfdf4459aae99c552cfba5b4b P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.009s
I20240501 08:12:39.348917 27891 consensus_queue.cc:237] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } }
I20240501 08:12:39.351195 28019 raft_consensus.cc:1261] T d6bb70263134414b9945a87087b55a2c P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:39.351882 30537 raft_consensus.cc:1261] T d6bb70263134414b9945a87087b55a2c P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:39.352304 817 consensus_queue.cc:1035] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:39.352854 584 raft_consensus.cc:1050] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6: attempting to promote NON_VOTER 31811708c6774037b82df44024b229b6 to VOTER
I20240501 08:12:39.353706 455 consensus_queue.cc:1035] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:39.358587 27604 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 489db424748b4a73966b047e81f36c83 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:39.358932 28019 raft_consensus.cc:2932] T d6bb70263134414b9945a87087b55a2c P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:39.359691 30537 raft_consensus.cc:2932] T d6bb70263134414b9945a87087b55a2c P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:39.361181 817 raft_consensus.cc:2932] T d6bb70263134414b9945a87087b55a2c P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:39.365324 887 raft_consensus.cc:348] T 28a7c90dfdf4459aae99c552cfba5b4b P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:12:39.365715 887 raft_consensus.cc:729] T 28a7c90dfdf4459aae99c552cfba5b4b P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: LEARNER
I20240501 08:12:39.366050 887 consensus_queue.cc:260] T 28a7c90dfdf4459aae99c552cfba5b4b P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:12:39.367033 27619 catalog_manager.cc:5484] T d6bb70263134414b9945a87087b55a2c P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:39.370481 27607 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d6bb70263134414b9945a87087b55a2c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:39.371410 887 ts_tablet_manager.cc:1398] T 28a7c90dfdf4459aae99c552cfba5b4b P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20240501 08:12:39.375576 584 consensus_queue.cc:237] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } }
I20240501 08:12:39.385171 30650 tablet_service.cc:1514] Processing DeleteTablet for tablet d6bb70263134414b9945a87087b55a2c with delete_type TABLET_DATA_TOMBSTONED (TS 9a022e1a501f4d44a0468054d6c3de47 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57156
I20240501 08:12:39.385991 28042 tablet_copy_service.cc:342] P aad81047009a4ffcbc450b60030583b5: Request end of tablet copy session a29cd590999b4f32849ff5689cd53c11-28a7c90dfdf4459aae99c552cfba5b4b received from {username='slave'} at 127.21.84.136:51325
I20240501 08:12:39.386127 28042 tablet_copy_service.cc:433] P aad81047009a4ffcbc450b60030583b5: ending tablet copy session a29cd590999b4f32849ff5689cd53c11-28a7c90dfdf4459aae99c552cfba5b4b on tablet 28a7c90dfdf4459aae99c552cfba5b4b with peer a29cd590999b4f32849ff5689cd53c11
I20240501 08:12:39.456516 905 ts_tablet_manager.cc:902] T 1448b8f2d5844ad7aec9bcd3874c5c01 P aad81047009a4ffcbc450b60030583b5: Initiating tablet copy from peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:39.462103 30537 raft_consensus.cc:1261] T e40fb0a0d49f44a1a36c33412c630077 P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:39.462252 27891 raft_consensus.cc:1261] T e40fb0a0d49f44a1a36c33412c630077 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:39.462752 28020 raft_consensus.cc:1261] T e40fb0a0d49f44a1a36c33412c630077 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:39.463368 32293 consensus_queue.cc:1035] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:39.464048 840 consensus_queue.cc:1035] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:39.467324 32293 consensus_queue.cc:1035] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:39.489180 32293 raft_consensus.cc:2932] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:39.500402 30537 raft_consensus.cc:2932] T e40fb0a0d49f44a1a36c33412c630077 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:39.502532 885 ts_tablet_manager.cc:1888] T b12559fc416a4f3cbffa018611226bcc P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:39.504261 885 log.cc:1196] T b12559fc416a4f3cbffa018611226bcc P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/b12559fc416a4f3cbffa018611226bcc
I20240501 08:12:39.505312 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet b12559fc416a4f3cbffa018611226bcc (table rebalance_test_table_17 [id=1121d4c7d5444060ac6df36744663e41]) successfully deleted
I20240501 08:12:39.507514 904 tablet_replica.cc:330] T d6bb70263134414b9945a87087b55a2c P 9a022e1a501f4d44a0468054d6c3de47: stopping tablet replica
I20240501 08:12:39.507825 904 raft_consensus.cc:2229] T d6bb70263134414b9945a87087b55a2c P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:39.508080 904 pending_rounds.cc:62] T d6bb70263134414b9945a87087b55a2c P 9a022e1a501f4d44a0468054d6c3de47: Trying to abort 1 pending ops.
I20240501 08:12:39.508205 904 pending_rounds.cc:68] T d6bb70263134414b9945a87087b55a2c P 9a022e1a501f4d44a0468054d6c3de47: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7022801548383494144 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "d6bb70263134414b9945a87087b55a2c" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } } }
I20240501 08:12:39.508767 904 raft_consensus.cc:2894] T d6bb70263134414b9945a87087b55a2c P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240501 08:12:39.508874 904 raft_consensus.cc:2866] T d6bb70263134414b9945a87087b55a2c P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 08:12:39.508965 904 raft_consensus.cc:2258] T d6bb70263134414b9945a87087b55a2c P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:39.518345 904 ts_tablet_manager.cc:1875] T d6bb70263134414b9945a87087b55a2c P 9a022e1a501f4d44a0468054d6c3de47: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:39.518918 28020 raft_consensus.cc:2932] T e40fb0a0d49f44a1a36c33412c630077 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:39.524137 27611 catalog_manager.cc:5484] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: config changed from index 4 to 5, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:39.530004 28149 consensus_queue.cc:237] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } }
I20240501 08:12:39.532932 904 ts_tablet_manager.cc:1888] T d6bb70263134414b9945a87087b55a2c P 9a022e1a501f4d44a0468054d6c3de47: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:39.533190 904 log.cc:1196] T d6bb70263134414b9945a87087b55a2c P 9a022e1a501f4d44a0468054d6c3de47: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/wals/d6bb70263134414b9945a87087b55a2c
I20240501 08:12:39.534271 27604 catalog_manager.cc:4830] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): tablet d6bb70263134414b9945a87087b55a2c (table rebalancer_extra_table_16 [id=8b4eeb92acc446a29303f629cf185156]) successfully deleted
I20240501 08:12:39.537331 30803 consensus_queue.cc:237] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
W20240501 08:12:39.539929 30740 consensus_peers.cc:485] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc9e0f35ebcd48b5a27c57b2485e599e. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:39.539914 30672 raft_consensus.cc:1261] T fc9e0f35ebcd48b5a27c57b2485e599e P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:39.540422 27761 raft_consensus.cc:1261] T fc9e0f35ebcd48b5a27c57b2485e599e P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:39.536334 30537 raft_consensus.cc:1261] T e40fb0a0d49f44a1a36c33412c630077 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:39.540963 32293 consensus_queue.cc:1035] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.541332 550 consensus_queue.cc:1035] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:39.542058 550 consensus_queue.cc:1035] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:39.545411 32293 raft_consensus.cc:2932] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:39.550277 27604 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e40fb0a0d49f44a1a36c33412c630077 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:39.551553 30534 raft_consensus.cc:2932] T e40fb0a0d49f44a1a36c33412c630077 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:39.558550 794 raft_consensus.cc:2932] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:39.559772 27611 catalog_manager.cc:5484] T e40fb0a0d49f44a1a36c33412c630077 P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index 5 to 6, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:39.568579 30935 raft_consensus.cc:1203] T 28a7c90dfdf4459aae99c552cfba5b4b P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:39.586436 28018 raft_consensus.cc:1261] T e40fb0a0d49f44a1a36c33412c630077 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:39.588198 307 consensus_queue.cc:1035] T e40fb0a0d49f44a1a36c33412c630077 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.590847 28018 raft_consensus.cc:2932] T e40fb0a0d49f44a1a36c33412c630077 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:39.599011 27761 raft_consensus.cc:2932] T fc9e0f35ebcd48b5a27c57b2485e599e P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:39.604194 27616 catalog_manager.cc:5484] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:39.619762 905 tablet_copy_client.cc:308] T 1448b8f2d5844ad7aec9bcd3874c5c01 P aad81047009a4ffcbc450b60030583b5: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.133:38293
I20240501 08:12:39.620676 30539 tablet_copy_service.cc:140] P 31811708c6774037b82df44024b229b6: Received BeginTabletCopySession request for tablet 1448b8f2d5844ad7aec9bcd3874c5c01 from peer aad81047009a4ffcbc450b60030583b5 ({username='slave'} at 127.21.84.131:56895)
I20240501 08:12:39.620908 30539 tablet_copy_service.cc:161] P 31811708c6774037b82df44024b229b6: Beginning new tablet copy session on tablet 1448b8f2d5844ad7aec9bcd3874c5c01 from peer aad81047009a4ffcbc450b60030583b5 at {username='slave'} at 127.21.84.131:56895: session id = aad81047009a4ffcbc450b60030583b5-1448b8f2d5844ad7aec9bcd3874c5c01
I20240501 08:12:39.629680 30539 tablet_copy_source_session.cc:205] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:39.630426 27871 tablet_service.cc:1514] Processing DeleteTablet for tablet e40fb0a0d49f44a1a36c33412c630077 with delete_type TABLET_DATA_TOMBSTONED (TS f1525cbc82a24d13b65f136695c903af not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:39.631875 905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1448b8f2d5844ad7aec9bcd3874c5c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:39.638399 905 tablet_copy_client.cc:783] T 1448b8f2d5844ad7aec9bcd3874c5c01 P aad81047009a4ffcbc450b60030583b5: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:39.638528 921 tablet_replica.cc:330] T e40fb0a0d49f44a1a36c33412c630077 P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:39.638793 905 tablet_copy_client.cc:647] T 1448b8f2d5844ad7aec9bcd3874c5c01 P aad81047009a4ffcbc450b60030583b5: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:39.638774 921 raft_consensus.cc:2229] T e40fb0a0d49f44a1a36c33412c630077 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:39.638749 28020 consensus_queue.cc:237] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:12:39.639115 921 pending_rounds.cc:62] T e40fb0a0d49f44a1a36c33412c630077 P f1525cbc82a24d13b65f136695c903af: Trying to abort 1 pending ops.
I20240501 08:12:39.639218 921 pending_rounds.cc:68] T e40fb0a0d49f44a1a36c33412c630077 P f1525cbc82a24d13b65f136695c903af: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7022801548799942656 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "e40fb0a0d49f44a1a36c33412c630077" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } } }
I20240501 08:12:39.639617 921 raft_consensus.cc:2894] T e40fb0a0d49f44a1a36c33412c630077 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240501 08:12:39.639725 921 raft_consensus.cc:2866] T e40fb0a0d49f44a1a36c33412c630077 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 08:12:39.639809 921 raft_consensus.cc:2258] T e40fb0a0d49f44a1a36c33412c630077 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:39.640884 28149 raft_consensus.cc:1261] T 5eccb9416d7d438893307f13dd4e73dc P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:39.640957 30803 raft_consensus.cc:1261] T 5eccb9416d7d438893307f13dd4e73dc P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:39.641189 921 ts_tablet_manager.cc:1875] T e40fb0a0d49f44a1a36c33412c630077 P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:39.641551 861 consensus_queue.cc:1035] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.643049 879 consensus_queue.cc:1035] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:39.643883 877 raft_consensus.cc:2932] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:39.644937 30803 raft_consensus.cc:2932] T 5eccb9416d7d438893307f13dd4e73dc P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:39.648492 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5eccb9416d7d438893307f13dd4e73dc with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:39.652180 28149 raft_consensus.cc:2932] T 5eccb9416d7d438893307f13dd4e73dc P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:39.653224 30669 raft_consensus.cc:2932] T fc9e0f35ebcd48b5a27c57b2485e599e P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:39.653384 27611 catalog_manager.cc:5484] T 5eccb9416d7d438893307f13dd4e73dc P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index 5 to 6, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:39.658021 27741 tablet_service.cc:1514] Processing DeleteTablet for tablet 5eccb9416d7d438893307f13dd4e73dc with delete_type TABLET_DATA_TOMBSTONED (TS 6331f77dae9d452184e8cd0cbf5f1b43 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:39.658573 896 tablet_replica.cc:330] T 5eccb9416d7d438893307f13dd4e73dc P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:39.658773 896 raft_consensus.cc:2229] T 5eccb9416d7d438893307f13dd4e73dc P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:39.659051 896 raft_consensus.cc:2258] T 5eccb9416d7d438893307f13dd4e73dc P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:39.661420 905 tablet_copy_client.cc:523] T 1448b8f2d5844ad7aec9bcd3874c5c01 P aad81047009a4ffcbc450b60030583b5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:39.666751 905 tablet_bootstrap.cc:492] T 1448b8f2d5844ad7aec9bcd3874c5c01 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:39.667326 896 ts_tablet_manager.cc:1875] T 5eccb9416d7d438893307f13dd4e73dc P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:39.675526 606 raft_consensus.cc:1050] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11: attempting to promote NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 to VOTER
I20240501 08:12:39.676407 606 consensus_queue.cc:237] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:12:39.681900 896 ts_tablet_manager.cc:1888] T 5eccb9416d7d438893307f13dd4e73dc P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:39.683045 896 log.cc:1196] T 5eccb9416d7d438893307f13dd4e73dc P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/5eccb9416d7d438893307f13dd4e73dc
I20240501 08:12:39.683982 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet 5eccb9416d7d438893307f13dd4e73dc (table rebalance_test_table_11 [id=05c446d8d19b4077b3441b0349357c6e]) successfully deleted
I20240501 08:12:39.686614 921 ts_tablet_manager.cc:1888] T e40fb0a0d49f44a1a36c33412c630077 P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:39.686919 921 log.cc:1196] T e40fb0a0d49f44a1a36c33412c630077 P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/e40fb0a0d49f44a1a36c33412c630077
I20240501 08:12:39.687853 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet e40fb0a0d49f44a1a36c33412c630077 (table rebalance_test_table_17 [id=1121d4c7d5444060ac6df36744663e41]) successfully deleted
I20240501 08:12:39.690541 28152 raft_consensus.cc:1261] T 6c324b1136ce429290d7286399bb7121 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:39.690889 905 tablet_bootstrap.cc:492] T 1448b8f2d5844ad7aec9bcd3874c5c01 P aad81047009a4ffcbc450b60030583b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:39.691092 456 consensus_queue.cc:1035] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:39.691516 905 tablet_bootstrap.cc:492] T 1448b8f2d5844ad7aec9bcd3874c5c01 P aad81047009a4ffcbc450b60030583b5: Bootstrap complete.
I20240501 08:12:39.691838 905 ts_tablet_manager.cc:1367] T 1448b8f2d5844ad7aec9bcd3874c5c01 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.002s
I20240501 08:12:39.692728 905 raft_consensus.cc:348] T 1448b8f2d5844ad7aec9bcd3874c5c01 P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:12:39.693064 905 raft_consensus.cc:729] T 1448b8f2d5844ad7aec9bcd3874c5c01 P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: LEARNER
I20240501 08:12:39.693410 905 consensus_queue.cc:260] T 1448b8f2d5844ad7aec9bcd3874c5c01 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:12:39.694315 905 ts_tablet_manager.cc:1398] T 1448b8f2d5844ad7aec9bcd3874c5c01 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:39.707098 30669 raft_consensus.cc:1261] T 6c324b1136ce429290d7286399bb7121 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:39.707682 456 consensus_queue.cc:1035] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:39.714908 30539 tablet_copy_service.cc:342] P 31811708c6774037b82df44024b229b6: Request end of tablet copy session aad81047009a4ffcbc450b60030583b5-1448b8f2d5844ad7aec9bcd3874c5c01 received from {username='slave'} at 127.21.84.131:56895
I20240501 08:12:39.715108 30539 tablet_copy_service.cc:433] P 31811708c6774037b82df44024b229b6: ending tablet copy session aad81047009a4ffcbc450b60030583b5-1448b8f2d5844ad7aec9bcd3874c5c01 on tablet 1448b8f2d5844ad7aec9bcd3874c5c01 with peer aad81047009a4ffcbc450b60030583b5
I20240501 08:12:39.721740 30803 raft_consensus.cc:1261] T 6c324b1136ce429290d7286399bb7121 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:39.726934 606 consensus_queue.cc:1035] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:39.732652 864 raft_consensus.cc:2932] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:39.737365 30669 raft_consensus.cc:2932] T 6c324b1136ce429290d7286399bb7121 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:39.741447 28152 raft_consensus.cc:2932] T 6c324b1136ce429290d7286399bb7121 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:39.742662 27619 catalog_manager.cc:5484] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:39.751065 30935 consensus_queue.cc:237] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:12:39.753612 28152 raft_consensus.cc:1261] T 6c324b1136ce429290d7286399bb7121 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:39.754106 30669 raft_consensus.cc:1261] T 6c324b1136ce429290d7286399bb7121 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:39.755036 32634 consensus_queue.cc:1035] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:39.755358 606 consensus_queue.cc:1035] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:39.758148 456 raft_consensus.cc:2932] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:39.760691 30803 raft_consensus.cc:2932] T 6c324b1136ce429290d7286399bb7121 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:39.761770 28149 raft_consensus.cc:2932] T 6c324b1136ce429290d7286399bb7121 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:39.770656 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6c324b1136ce429290d7286399bb7121 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:39.770812 27611 catalog_manager.cc:5484] T 6c324b1136ce429290d7286399bb7121 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: config changed from index 3 to 4, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:39.773675 30669 raft_consensus.cc:2932] T 6c324b1136ce429290d7286399bb7121 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:39.780562 30784 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c324b1136ce429290d7286399bb7121 with delete_type TABLET_DATA_TOMBSTONED (TS 64998323a1f4482d8704bf20594d13a5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38952
I20240501 08:12:39.795343 931 tablet_replica.cc:330] T 6c324b1136ce429290d7286399bb7121 P 64998323a1f4482d8704bf20594d13a5: stopping tablet replica
I20240501 08:12:39.795778 931 raft_consensus.cc:2229] T 6c324b1136ce429290d7286399bb7121 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:39.796204 931 raft_consensus.cc:2258] T 6c324b1136ce429290d7286399bb7121 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:39.798159 931 ts_tablet_manager.cc:1875] T 6c324b1136ce429290d7286399bb7121 P 64998323a1f4482d8704bf20594d13a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:39.809628 931 ts_tablet_manager.cc:1888] T 6c324b1136ce429290d7286399bb7121 P 64998323a1f4482d8704bf20594d13a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:39.809962 931 log.cc:1196] T 6c324b1136ce429290d7286399bb7121 P 64998323a1f4482d8704bf20594d13a5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal/wals/6c324b1136ce429290d7286399bb7121
I20240501 08:12:39.810863 27605 catalog_manager.cc:4830] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): tablet 6c324b1136ce429290d7286399bb7121 (table rebalancer_extra_table_46 [id=ec67e56c6aaa43f3b2e573534276d8bc]) successfully deleted
I20240501 08:12:39.961715 28020 raft_consensus.cc:1203] T 1448b8f2d5844ad7aec9bcd3874c5c01 P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:39.974324 984 ts_tablet_manager.cc:902] T fc9e0f35ebcd48b5a27c57b2485e599e P f1525cbc82a24d13b65f136695c903af: Initiating tablet copy from peer 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:39.977389 984 tablet_copy_client.cc:308] T fc9e0f35ebcd48b5a27c57b2485e599e P f1525cbc82a24d13b65f136695c903af: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.135:33503
I20240501 08:12:39.978204 30825 tablet_copy_service.cc:140] P 64998323a1f4482d8704bf20594d13a5: Received BeginTabletCopySession request for tablet fc9e0f35ebcd48b5a27c57b2485e599e from peer f1525cbc82a24d13b65f136695c903af ({username='slave'} at 127.21.84.130:45079)
I20240501 08:12:39.978376 30825 tablet_copy_service.cc:161] P 64998323a1f4482d8704bf20594d13a5: Beginning new tablet copy session on tablet fc9e0f35ebcd48b5a27c57b2485e599e from peer f1525cbc82a24d13b65f136695c903af at {username='slave'} at 127.21.84.130:45079: session id = f1525cbc82a24d13b65f136695c903af-fc9e0f35ebcd48b5a27c57b2485e599e
I20240501 08:12:39.981004 30825 tablet_copy_source_session.cc:205] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:39.982894 984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc9e0f35ebcd48b5a27c57b2485e599e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:39.987663 984 tablet_copy_client.cc:783] T fc9e0f35ebcd48b5a27c57b2485e599e P f1525cbc82a24d13b65f136695c903af: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:39.987970 984 tablet_copy_client.cc:647] T fc9e0f35ebcd48b5a27c57b2485e599e P f1525cbc82a24d13b65f136695c903af: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:39.990370 984 tablet_copy_client.cc:523] T fc9e0f35ebcd48b5a27c57b2485e599e P f1525cbc82a24d13b65f136695c903af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:39.995180 984 tablet_bootstrap.cc:492] T fc9e0f35ebcd48b5a27c57b2485e599e P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:40.007145 984 tablet_bootstrap.cc:492] T fc9e0f35ebcd48b5a27c57b2485e599e P f1525cbc82a24d13b65f136695c903af: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:40.007628 984 tablet_bootstrap.cc:492] T fc9e0f35ebcd48b5a27c57b2485e599e P f1525cbc82a24d13b65f136695c903af: Bootstrap complete.
I20240501 08:12:40.007938 984 ts_tablet_manager.cc:1367] T fc9e0f35ebcd48b5a27c57b2485e599e P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.006s
I20240501 08:12:40.010162 984 raft_consensus.cc:348] T fc9e0f35ebcd48b5a27c57b2485e599e P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:40.010393 984 raft_consensus.cc:729] T fc9e0f35ebcd48b5a27c57b2485e599e P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: LEARNER
I20240501 08:12:40.010555 984 consensus_queue.cc:260] T fc9e0f35ebcd48b5a27c57b2485e599e P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:40.011101 984 ts_tablet_manager.cc:1398] T fc9e0f35ebcd48b5a27c57b2485e599e P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 08:12:40.012617 30825 tablet_copy_service.cc:342] P 64998323a1f4482d8704bf20594d13a5: Request end of tablet copy session f1525cbc82a24d13b65f136695c903af-fc9e0f35ebcd48b5a27c57b2485e599e received from {username='slave'} at 127.21.84.130:45079
I20240501 08:12:40.013366 30825 tablet_copy_service.cc:433] P 64998323a1f4482d8704bf20594d13a5: ending tablet copy session f1525cbc82a24d13b65f136695c903af-fc9e0f35ebcd48b5a27c57b2485e599e on tablet fc9e0f35ebcd48b5a27c57b2485e599e with peer f1525cbc82a24d13b65f136695c903af
I20240501 08:12:40.127395 879 raft_consensus.cc:1050] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5: attempting to promote NON_VOTER a29cd590999b4f32849ff5689cd53c11 to VOTER
I20240501 08:12:40.128054 879 consensus_queue.cc:237] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:12:40.144199 30534 raft_consensus.cc:1261] T 28a7c90dfdf4459aae99c552cfba5b4b P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:40.144197 30803 raft_consensus.cc:1261] T 28a7c90dfdf4459aae99c552cfba5b4b P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:40.144266 30935 raft_consensus.cc:1261] T 28a7c90dfdf4459aae99c552cfba5b4b P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:40.144901 878 consensus_queue.cc:1035] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:40.145610 879 consensus_queue.cc:1035] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:40.145982 861 consensus_queue.cc:1035] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:40.197227 861 raft_consensus.cc:2932] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:40.197814 30803 raft_consensus.cc:2932] T 28a7c90dfdf4459aae99c552cfba5b4b P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:40.198891 30534 raft_consensus.cc:2932] T 28a7c90dfdf4459aae99c552cfba5b4b P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:40.204733 30935 raft_consensus.cc:2932] T 28a7c90dfdf4459aae99c552cfba5b4b P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:40.207258 27619 catalog_manager.cc:5484] T 28a7c90dfdf4459aae99c552cfba5b4b P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index 2 to 3, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:40.220244 28020 consensus_queue.cc:237] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:12:40.222291 30804 raft_consensus.cc:1261] T 28a7c90dfdf4459aae99c552cfba5b4b P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:40.222798 861 consensus_queue.cc:1035] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:40.223199 30937 raft_consensus.cc:1261] T 28a7c90dfdf4459aae99c552cfba5b4b P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:40.224296 27761 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8ab0410a98974002ab68dd0f6a456d59"
dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59992
I20240501 08:12:40.224550 27761 raft_consensus.cc:595] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:12:40.228998 878 consensus_queue.cc:1035] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:40.229920 878 raft_consensus.cc:2932] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:40.233263 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 28a7c90dfdf4459aae99c552cfba5b4b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:40.235145 30804 raft_consensus.cc:2932] T 28a7c90dfdf4459aae99c552cfba5b4b P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:40.240262 30935 raft_consensus.cc:2932] T 28a7c90dfdf4459aae99c552cfba5b4b P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:40.241240 27619 catalog_manager.cc:5484] T 28a7c90dfdf4459aae99c552cfba5b4b P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:40.247437 30514 tablet_service.cc:1514] Processing DeleteTablet for tablet 28a7c90dfdf4459aae99c552cfba5b4b with delete_type TABLET_DATA_TOMBSTONED (TS 31811708c6774037b82df44024b229b6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45078
I20240501 08:12:40.248307 1043 tablet_replica.cc:330] T 28a7c90dfdf4459aae99c552cfba5b4b P 31811708c6774037b82df44024b229b6: stopping tablet replica
I20240501 08:12:40.248541 1043 raft_consensus.cc:2229] T 28a7c90dfdf4459aae99c552cfba5b4b P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:40.248843 1043 raft_consensus.cc:2258] T 28a7c90dfdf4459aae99c552cfba5b4b P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:40.250100 1043 ts_tablet_manager.cc:1875] T 28a7c90dfdf4459aae99c552cfba5b4b P 31811708c6774037b82df44024b229b6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:40.257021 1043 ts_tablet_manager.cc:1888] T 28a7c90dfdf4459aae99c552cfba5b4b P 31811708c6774037b82df44024b229b6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:40.257210 1043 log.cc:1196] T 28a7c90dfdf4459aae99c552cfba5b4b P 31811708c6774037b82df44024b229b6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/wals/28a7c90dfdf4459aae99c552cfba5b4b
I20240501 08:12:40.258116 27607 catalog_manager.cc:4830] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): tablet 28a7c90dfdf4459aae99c552cfba5b4b (table rebalancer_extra_table_6 [id=2b1f439c36b24b19bec52d25077c330d]) successfully deleted
I20240501 08:12:40.294540 28020 consensus_queue.cc:237] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
I20240501 08:12:40.297031 28152 raft_consensus.cc:1261] T 6c04880ea87d4894bd3a63d94f4a69f4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:40.297117 27761 raft_consensus.cc:1261] T 6c04880ea87d4894bd3a63d94f4a69f4 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:40.297549 861 consensus_queue.cc:1035] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:12:40.297919 27958 consensus_peers.cc:485] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 -> Peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): Couldn't send request to peer 31811708c6774037b82df44024b229b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c04880ea87d4894bd3a63d94f4a69f4. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:40.298396 861 consensus_queue.cc:1035] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:40.302708 877 raft_consensus.cc:2932] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:12:40.303195 27761 raft_consensus.cc:2932] T 6c04880ea87d4894bd3a63d94f4a69f4 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:12:40.310006 28152 raft_consensus.cc:2932] T 6c04880ea87d4894bd3a63d94f4a69f4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:12:40.314916 27619 catalog_manager.cc:5484] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index -1 to 4, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:40.336232 28152 consensus_queue.cc:237] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:12:40.338376 27891 raft_consensus.cc:1261] T 05b2485fa01a4706a85ebc150306637f P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:40.338534 28018 raft_consensus.cc:1261] T 05b2485fa01a4706a85ebc150306637f P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:12:40.338651 28087 consensus_peers.cc:485] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6 -> Peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): Couldn't send request to peer a29cd590999b4f32849ff5689cd53c11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05b2485fa01a4706a85ebc150306637f. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:40.339002 32293 consensus_queue.cc:1035] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:40.339253 307 consensus_queue.cc:1035] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:40.342445 840 raft_consensus.cc:2932] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:40.343014 27891 raft_consensus.cc:2932] T 05b2485fa01a4706a85ebc150306637f P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:40.343052 28018 raft_consensus.cc:2932] T 05b2485fa01a4706a85ebc150306637f P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:40.348248 27611 catalog_manager.cc:5484] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: config changed from index -1 to 4, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:40.370276 27891 consensus_queue.cc:237] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
I20240501 08:12:40.373157 27761 raft_consensus.cc:1261] T 0be6dad1a409426c9a3dc0866505b3ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:40.373872 810 consensus_queue.cc:1035] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:12:40.374624 27826 consensus_peers.cc:485] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af -> Peer 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): Couldn't send request to peer 64998323a1f4482d8704bf20594d13a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0be6dad1a409426c9a3dc0866505b3ea. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:40.387883 28020 raft_consensus.cc:1261] T 0be6dad1a409426c9a3dc0866505b3ea P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:40.389400 523 raft_consensus.cc:1050] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6: attempting to promote NON_VOTER aad81047009a4ffcbc450b60030583b5 to VOTER
I20240501 08:12:40.390611 523 consensus_queue.cc:237] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } }
I20240501 08:12:40.393548 28018 raft_consensus.cc:1261] T 1448b8f2d5844ad7aec9bcd3874c5c01 P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:40.394165 30669 raft_consensus.cc:1261] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:40.395076 939 consensus_queue.cc:1035] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:40.396737 30935 raft_consensus.cc:1261] T 1448b8f2d5844ad7aec9bcd3874c5c01 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:40.397141 983 consensus_queue.cc:1035] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:40.398074 983 consensus_queue.cc:1035] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:40.404450 1070 consensus_queue.cc:1035] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:40.405162 1070 raft_consensus.cc:2932] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:12:40.412796 28020 raft_consensus.cc:2932] T 0be6dad1a409426c9a3dc0866505b3ea P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:12:40.414317 983 raft_consensus.cc:2932] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:40.422638 27611 catalog_manager.cc:5484] T 0be6dad1a409426c9a3dc0866505b3ea P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index -1 to 4, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:12:40.423537 28018 raft_consensus.cc:2932] T 1448b8f2d5844ad7aec9bcd3874c5c01 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:40.429255 27761 raft_consensus.cc:2932] T 0be6dad1a409426c9a3dc0866505b3ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:12:40.440629 30669 raft_consensus.cc:2932] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:40.443025 30937 raft_consensus.cc:2932] T 1448b8f2d5844ad7aec9bcd3874c5c01 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:40.443841 27611 catalog_manager.cc:5484] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:40.491408 30534 consensus_queue.cc:237] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } }
I20240501 08:12:40.496428 28019 raft_consensus.cc:1261] T 1448b8f2d5844ad7aec9bcd3874c5c01 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:40.497032 939 consensus_queue.cc:1035] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:40.502046 30937 raft_consensus.cc:1261] T 1448b8f2d5844ad7aec9bcd3874c5c01 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:40.501662 939 raft_consensus.cc:2932] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:40.503966 1073 consensus_queue.cc:1035] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:40.507887 27607 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1448b8f2d5844ad7aec9bcd3874c5c01 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:40.508772 28019 raft_consensus.cc:2932] T 1448b8f2d5844ad7aec9bcd3874c5c01 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:40.509588 30935 raft_consensus.cc:2932] T 1448b8f2d5844ad7aec9bcd3874c5c01 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:40.515928 27611 catalog_manager.cc:5484] T 1448b8f2d5844ad7aec9bcd3874c5c01 P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:40.520310 30650 tablet_service.cc:1514] Processing DeleteTablet for tablet 1448b8f2d5844ad7aec9bcd3874c5c01 with delete_type TABLET_DATA_TOMBSTONED (TS 9a022e1a501f4d44a0468054d6c3de47 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57156
I20240501 08:12:40.521083 1091 tablet_replica.cc:330] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 9a022e1a501f4d44a0468054d6c3de47: stopping tablet replica
I20240501 08:12:40.521342 1091 raft_consensus.cc:2229] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:40.521638 1091 raft_consensus.cc:2258] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:40.523900 1091 ts_tablet_manager.cc:1875] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 9a022e1a501f4d44a0468054d6c3de47: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:40.529498 1089 raft_consensus.cc:979] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43: : Instructing follower a29cd590999b4f32849ff5689cd53c11 to start an election
I20240501 08:12:40.529654 1089 raft_consensus.cc:1067] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Signalling peer a29cd590999b4f32849ff5689cd53c11 to start an election
I20240501 08:12:40.531016 30935 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8ab0410a98974002ab68dd0f6a456d59"
dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
from {username='slave'} at 127.21.84.129:48739
I20240501 08:12:40.531312 30935 raft_consensus.cc:482] T 8ab0410a98974002ab68dd0f6a456d59 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:40.531443 30935 raft_consensus.cc:3037] T 8ab0410a98974002ab68dd0f6a456d59 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:40.532313 1091 ts_tablet_manager.cc:1888] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 9a022e1a501f4d44a0468054d6c3de47: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:40.532516 1091 log.cc:1196] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 9a022e1a501f4d44a0468054d6c3de47: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/wals/1448b8f2d5844ad7aec9bcd3874c5c01
I20240501 08:12:40.533438 27604 catalog_manager.cc:4830] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): tablet 1448b8f2d5844ad7aec9bcd3874c5c01 (table rebalancer_extra_table_10 [id=6377dbf65f184f9cbe44f12bd6a0405f]) successfully deleted
I20240501 08:12:40.534317 30935 raft_consensus.cc:504] T 8ab0410a98974002ab68dd0f6a456d59 P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:12:40.535234 30935 leader_election.cc:288] T 8ab0410a98974002ab68dd0f6a456d59 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 election: Requested vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:40.535641 28152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab0410a98974002ab68dd0f6a456d59" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:12:40.535676 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab0410a98974002ab68dd0f6a456d59" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:12:40.535856 28152 raft_consensus.cc:3037] T 8ab0410a98974002ab68dd0f6a456d59 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:40.535856 27761 raft_consensus.cc:3032] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:12:40.535939 27761 raft_consensus.cc:729] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:12:40.536299 27761 consensus_queue.cc:260] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:12:40.536772 27761 raft_consensus.cc:3037] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:40.538560 28152 raft_consensus.cc:2454] T 8ab0410a98974002ab68dd0f6a456d59 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 2.
I20240501 08:12:40.539371 27761 raft_consensus.cc:2454] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 2.
I20240501 08:12:40.539960 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab0410a98974002ab68dd0f6a456d59" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:40.540191 30804 raft_consensus.cc:3037] T 8ab0410a98974002ab68dd0f6a456d59 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:40.542536 30870 leader_election.cc:302] T 8ab0410a98974002ab68dd0f6a456d59 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 91d11eae5ef6432ca86eb8ed446b6ea6, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:40.542953 456 raft_consensus.cc:2781] T 8ab0410a98974002ab68dd0f6a456d59 P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:12:40.543155 456 raft_consensus.cc:686] T 8ab0410a98974002ab68dd0f6a456d59 P a29cd590999b4f32849ff5689cd53c11 [term 2 LEADER]: Becoming Leader. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:12:40.543254 30804 raft_consensus.cc:2454] T 8ab0410a98974002ab68dd0f6a456d59 P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 2.
I20240501 08:12:40.543473 456 consensus_queue.cc:237] T 8ab0410a98974002ab68dd0f6a456d59 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:12:40.556887 27619 catalog_manager.cc:5484] T 8ab0410a98974002ab68dd0f6a456d59 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: term changed from 1 to 2, leader changed from 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) to a29cd590999b4f32849ff5689cd53c11 (127.21.84.136). New cstate: current_term: 2 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:40.568529 30937 consensus_queue.cc:237] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:12:40.571570 30669 raft_consensus.cc:1261] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:40.571641 27761 raft_consensus.cc:1261] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:12:40.572026 30870 consensus_peers.cc:485] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 -> Peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Couldn't send request to peer 91d11eae5ef6432ca86eb8ed446b6ea6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8bbd50f618d14aad8e45c2aa6fbabc1f. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:40.572392 864 consensus_queue.cc:1035] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:40.573001 606 consensus_queue.cc:1035] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:40.585590 27891 raft_consensus.cc:1203] T fc9e0f35ebcd48b5a27c57b2485e599e P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:40.594784 606 raft_consensus.cc:2932] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:12:40.603224 27760 raft_consensus.cc:2932] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:12:40.613847 27619 catalog_manager.cc:5484] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:40.612135 30672 raft_consensus.cc:2932] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:12:40.654229 30672 consensus_queue.cc:237] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
I20240501 08:12:40.657193 30937 raft_consensus.cc:1261] T 448a24d99bbe4fe4a571dc49a04ee275 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:40.657212 27760 raft_consensus.cc:1261] T 448a24d99bbe4fe4a571dc49a04ee275 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:12:40.658118 30608 consensus_peers.cc:485] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 -> Peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): Couldn't send request to peer 31811708c6774037b82df44024b229b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 448a24d99bbe4fe4a571dc49a04ee275. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:40.658967 362 consensus_queue.cc:1035] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:40.659302 1108 consensus_queue.cc:1035] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:40.663684 850 raft_consensus.cc:2932] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:12:40.664584 27760 raft_consensus.cc:2932] T 448a24d99bbe4fe4a571dc49a04ee275 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:12:40.664589 30935 raft_consensus.cc:2932] T 448a24d99bbe4fe4a571dc49a04ee275 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:12:40.671106 27619 catalog_manager.cc:5484] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index -1 to 2, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New cstate: current_term: 1 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:40.701341 30534 consensus_queue.cc:237] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:12:40.705804 28152 raft_consensus.cc:1261] T d020e1225d5d447f993f920569a05f3d P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:40.706270 27891 raft_consensus.cc:1261] T d020e1225d5d447f993f920569a05f3d P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:12:40.706296 30472 consensus_peers.cc:485] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 -> Peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): Couldn't send request to peer a29cd590999b4f32849ff5689cd53c11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d020e1225d5d447f993f920569a05f3d. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:40.706423 939 consensus_queue.cc:1035] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:40.707021 983 consensus_queue.cc:1035] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:40.711637 983 raft_consensus.cc:2932] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:40.715083 27891 raft_consensus.cc:2932] T d020e1225d5d447f993f920569a05f3d P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:40.724831 27619 catalog_manager.cc:5484] T d020e1225d5d447f993f920569a05f3d P f1525cbc82a24d13b65f136695c903af reported cstate change: config changed from index -1 to 2, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:40.731055 28152 raft_consensus.cc:2932] T d020e1225d5d447f993f920569a05f3d P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:40.752943 1123 ts_tablet_manager.cc:902] T 6c04880ea87d4894bd3a63d94f4a69f4 P 31811708c6774037b82df44024b229b6: Initiating tablet copy from peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:40.778363 1123 tablet_copy_client.cc:308] T 6c04880ea87d4894bd3a63d94f4a69f4 P 31811708c6774037b82df44024b229b6: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.131:36305
I20240501 08:12:40.779075 28042 tablet_copy_service.cc:140] P aad81047009a4ffcbc450b60030583b5: Received BeginTabletCopySession request for tablet 6c04880ea87d4894bd3a63d94f4a69f4 from peer 31811708c6774037b82df44024b229b6 ({username='slave'} at 127.21.84.133:53413)
I20240501 08:12:40.779235 28042 tablet_copy_service.cc:161] P aad81047009a4ffcbc450b60030583b5: Beginning new tablet copy session on tablet 6c04880ea87d4894bd3a63d94f4a69f4 from peer 31811708c6774037b82df44024b229b6 at {username='slave'} at 127.21.84.133:53413: session id = 31811708c6774037b82df44024b229b6-6c04880ea87d4894bd3a63d94f4a69f4
I20240501 08:12:40.783643 28042 tablet_copy_source_session.cc:205] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:40.785096 1123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c04880ea87d4894bd3a63d94f4a69f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:40.785202 1132 ts_tablet_manager.cc:902] T 0be6dad1a409426c9a3dc0866505b3ea P 64998323a1f4482d8704bf20594d13a5: Initiating tablet copy from peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:12:40.787631 27760 consensus_queue.cc:237] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
I20240501 08:12:40.790457 27891 raft_consensus.cc:1261] T 32a54b70dcc447828ec293a9ff0afb80 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:40.790649 28019 raft_consensus.cc:1261] T 32a54b70dcc447828ec293a9ff0afb80 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:40.791345 1089 consensus_queue.cc:1035] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:40.791883 1087 consensus_queue.cc:1035] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:12:40.792172 27695 consensus_peers.cc:485] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 -> Peer 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): Couldn't send request to peer 64998323a1f4482d8704bf20594d13a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32a54b70dcc447828ec293a9ff0afb80. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:40.794723 1089 raft_consensus.cc:2932] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:12:40.795430 27891 raft_consensus.cc:2932] T 32a54b70dcc447828ec293a9ff0afb80 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:12:40.799611 1132 tablet_copy_client.cc:308] T 0be6dad1a409426c9a3dc0866505b3ea P 64998323a1f4482d8704bf20594d13a5: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.130:36503
I20240501 08:12:40.800269 27912 tablet_copy_service.cc:140] P f1525cbc82a24d13b65f136695c903af: Received BeginTabletCopySession request for tablet 0be6dad1a409426c9a3dc0866505b3ea from peer 64998323a1f4482d8704bf20594d13a5 ({username='slave'} at 127.21.84.135:48367)
I20240501 08:12:40.800410 27912 tablet_copy_service.cc:161] P f1525cbc82a24d13b65f136695c903af: Beginning new tablet copy session on tablet 0be6dad1a409426c9a3dc0866505b3ea from peer 64998323a1f4482d8704bf20594d13a5 at {username='slave'} at 127.21.84.135:48367: session id = 64998323a1f4482d8704bf20594d13a5-0be6dad1a409426c9a3dc0866505b3ea
I20240501 08:12:40.803609 27912 tablet_copy_source_session.cc:205] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:40.807046 1132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0be6dad1a409426c9a3dc0866505b3ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:40.808034 27612 catalog_manager.cc:5484] T 32a54b70dcc447828ec293a9ff0afb80 P f1525cbc82a24d13b65f136695c903af reported cstate change: config changed from index -1 to 4, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:12:40.813372 1123 tablet_copy_client.cc:783] T 6c04880ea87d4894bd3a63d94f4a69f4 P 31811708c6774037b82df44024b229b6: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:40.813859 1123 tablet_copy_client.cc:647] T 6c04880ea87d4894bd3a63d94f4a69f4 P 31811708c6774037b82df44024b229b6: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:40.815964 1137 ts_tablet_manager.cc:902] T 05b2485fa01a4706a85ebc150306637f P a29cd590999b4f32849ff5689cd53c11: Initiating tablet copy from peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:40.817838 1123 tablet_copy_client.cc:523] T 6c04880ea87d4894bd3a63d94f4a69f4 P 31811708c6774037b82df44024b229b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:40.818362 28019 raft_consensus.cc:2932] T 32a54b70dcc447828ec293a9ff0afb80 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:12:40.822794 1123 tablet_bootstrap.cc:492] T 6c04880ea87d4894bd3a63d94f4a69f4 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:40.834403 1132 tablet_copy_client.cc:783] T 0be6dad1a409426c9a3dc0866505b3ea P 64998323a1f4482d8704bf20594d13a5: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:40.835088 1132 tablet_copy_client.cc:647] T 0be6dad1a409426c9a3dc0866505b3ea P 64998323a1f4482d8704bf20594d13a5: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:40.843322 1132 tablet_copy_client.cc:523] T 0be6dad1a409426c9a3dc0866505b3ea P 64998323a1f4482d8704bf20594d13a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:40.849130 1132 tablet_bootstrap.cc:492] T 0be6dad1a409426c9a3dc0866505b3ea P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:40.854490 1137 tablet_copy_client.cc:308] T 05b2485fa01a4706a85ebc150306637f P a29cd590999b4f32849ff5689cd53c11: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.132:41323
I20240501 08:12:40.855252 28172 tablet_copy_service.cc:140] P 91d11eae5ef6432ca86eb8ed446b6ea6: Received BeginTabletCopySession request for tablet 05b2485fa01a4706a85ebc150306637f from peer a29cd590999b4f32849ff5689cd53c11 ({username='slave'} at 127.21.84.136:39335)
I20240501 08:12:40.855392 28172 tablet_copy_service.cc:161] P 91d11eae5ef6432ca86eb8ed446b6ea6: Beginning new tablet copy session on tablet 05b2485fa01a4706a85ebc150306637f from peer a29cd590999b4f32849ff5689cd53c11 at {username='slave'} at 127.21.84.136:39335: session id = a29cd590999b4f32849ff5689cd53c11-05b2485fa01a4706a85ebc150306637f
I20240501 08:12:40.857975 28172 tablet_copy_source_session.cc:205] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:40.863723 1137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05b2485fa01a4706a85ebc150306637f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:40.877820 1137 tablet_copy_client.cc:783] T 05b2485fa01a4706a85ebc150306637f P a29cd590999b4f32849ff5689cd53c11: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:40.878754 1137 tablet_copy_client.cc:647] T 05b2485fa01a4706a85ebc150306637f P a29cd590999b4f32849ff5689cd53c11: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:40.885983 30804 consensus_queue.cc:237] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:12:40.909925 1137 tablet_copy_client.cc:523] T 05b2485fa01a4706a85ebc150306637f P a29cd590999b4f32849ff5689cd53c11: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:40.912467 1123 tablet_bootstrap.cc:492] T 6c04880ea87d4894bd3a63d94f4a69f4 P 31811708c6774037b82df44024b229b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:40.913035 1123 tablet_bootstrap.cc:492] T 6c04880ea87d4894bd3a63d94f4a69f4 P 31811708c6774037b82df44024b229b6: Bootstrap complete.
I20240501 08:12:40.913338 1123 ts_tablet_manager.cc:1367] T 6c04880ea87d4894bd3a63d94f4a69f4 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.091s user 0.054s sys 0.017s
I20240501 08:12:40.914426 1137 tablet_bootstrap.cc:492] T 05b2485fa01a4706a85ebc150306637f P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:40.914376 1123 raft_consensus.cc:348] T 6c04880ea87d4894bd3a63d94f4a69f4 P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
I20240501 08:12:40.914674 1123 raft_consensus.cc:729] T 6c04880ea87d4894bd3a63d94f4a69f4 P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: LEARNER
I20240501 08:12:40.914897 1123 consensus_queue.cc:260] T 6c04880ea87d4894bd3a63d94f4a69f4 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
I20240501 08:12:40.915549 1123 ts_tablet_manager.cc:1398] T 6c04880ea87d4894bd3a63d94f4a69f4 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
I20240501 08:12:40.916935 28042 tablet_copy_service.cc:342] P aad81047009a4ffcbc450b60030583b5: Request end of tablet copy session 31811708c6774037b82df44024b229b6-6c04880ea87d4894bd3a63d94f4a69f4 received from {username='slave'} at 127.21.84.133:53413
I20240501 08:12:40.917054 28042 tablet_copy_service.cc:433] P aad81047009a4ffcbc450b60030583b5: ending tablet copy session 31811708c6774037b82df44024b229b6-6c04880ea87d4894bd3a63d94f4a69f4 on tablet 6c04880ea87d4894bd3a63d94f4a69f4 with peer 31811708c6774037b82df44024b229b6
I20240501 08:12:40.921736 28019 raft_consensus.cc:1261] T 2e1dd7a48a574ffcbea936af5b0fe192 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:40.925282 915 consensus_queue.cc:1035] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
W20240501 08:12:40.925917 30737 consensus_peers.cc:485] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 -> Peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Couldn't send request to peer 9a022e1a501f4d44a0468054d6c3de47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e1dd7a48a574ffcbea936af5b0fe192. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:40.935755 28149 raft_consensus.cc:1261] T 2e1dd7a48a574ffcbea936af5b0fe192 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:40.937569 915 consensus_queue.cc:1035] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:40.938000 1132 tablet_bootstrap.cc:492] T 0be6dad1a409426c9a3dc0866505b3ea P 64998323a1f4482d8704bf20594d13a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:40.938519 1132 tablet_bootstrap.cc:492] T 0be6dad1a409426c9a3dc0866505b3ea P 64998323a1f4482d8704bf20594d13a5: Bootstrap complete.
I20240501 08:12:40.938892 1132 ts_tablet_manager.cc:1367] T 0be6dad1a409426c9a3dc0866505b3ea P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.090s user 0.055s sys 0.022s
I20240501 08:12:40.939792 1132 raft_consensus.cc:348] T 0be6dad1a409426c9a3dc0866505b3ea P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
I20240501 08:12:40.940155 1132 raft_consensus.cc:729] T 0be6dad1a409426c9a3dc0866505b3ea P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: LEARNER
I20240501 08:12:40.940407 1132 consensus_queue.cc:260] T 0be6dad1a409426c9a3dc0866505b3ea P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
I20240501 08:12:40.942546 1132 ts_tablet_manager.cc:1398] T 0be6dad1a409426c9a3dc0866505b3ea P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:12:40.946316 27912 tablet_copy_service.cc:342] P f1525cbc82a24d13b65f136695c903af: Request end of tablet copy session 64998323a1f4482d8704bf20594d13a5-0be6dad1a409426c9a3dc0866505b3ea received from {username='slave'} at 127.21.84.135:48367
I20240501 08:12:40.946471 27912 tablet_copy_service.cc:433] P f1525cbc82a24d13b65f136695c903af: ending tablet copy session 64998323a1f4482d8704bf20594d13a5-0be6dad1a409426c9a3dc0866505b3ea on tablet 0be6dad1a409426c9a3dc0866505b3ea with peer 64998323a1f4482d8704bf20594d13a5
I20240501 08:12:40.956182 915 raft_consensus.cc:2932] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:40.960492 28019 raft_consensus.cc:2932] T 2e1dd7a48a574ffcbea936af5b0fe192 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:40.969211 27619 catalog_manager.cc:5484] T 2e1dd7a48a574ffcbea936af5b0fe192 P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index -1 to 2, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:40.969310 28151 raft_consensus.cc:2932] T 2e1dd7a48a574ffcbea936af5b0fe192 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:40.979350 1154 raft_consensus.cc:1050] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5: attempting to promote NON_VOTER f1525cbc82a24d13b65f136695c903af to VOTER
I20240501 08:12:40.980257 1154 consensus_queue.cc:237] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:40.983009 27761 raft_consensus.cc:1261] T fc9e0f35ebcd48b5a27c57b2485e599e P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:40.983008 27891 raft_consensus.cc:1261] T fc9e0f35ebcd48b5a27c57b2485e599e P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:40.983698 915 consensus_queue.cc:1035] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:40.984105 793 consensus_queue.cc:1035] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:40.984373 30669 raft_consensus.cc:1261] T fc9e0f35ebcd48b5a27c57b2485e599e P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:40.984936 793 consensus_queue.cc:1035] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:40.987752 915 raft_consensus.cc:2932] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:40.993142 27761 raft_consensus.cc:2932] T fc9e0f35ebcd48b5a27c57b2485e599e P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:40.997766 27891 raft_consensus.cc:2932] T fc9e0f35ebcd48b5a27c57b2485e599e P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:41.001564 27619 catalog_manager.cc:5484] T fc9e0f35ebcd48b5a27c57b2485e599e P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:41.017125 30672 raft_consensus.cc:2932] T fc9e0f35ebcd48b5a27c57b2485e599e P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:41.028110 30804 consensus_queue.cc:237] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:41.030681 30672 raft_consensus.cc:1261] T fc9e0f35ebcd48b5a27c57b2485e599e P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:41.030691 27891 raft_consensus.cc:1261] T fc9e0f35ebcd48b5a27c57b2485e599e P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:41.031522 915 consensus_queue.cc:1035] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:41.031888 1160 consensus_queue.cc:1035] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:41.035609 1154 raft_consensus.cc:2932] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:41.036444 27891 raft_consensus.cc:2932] T fc9e0f35ebcd48b5a27c57b2485e599e P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:41.041332 27605 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fc9e0f35ebcd48b5a27c57b2485e599e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:41.041689 30669 raft_consensus.cc:2932] T fc9e0f35ebcd48b5a27c57b2485e599e P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:41.047997 27619 catalog_manager.cc:5484] T fc9e0f35ebcd48b5a27c57b2485e599e P f1525cbc82a24d13b65f136695c903af reported cstate change: config changed from index 3 to 4, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:41.051821 30937 consensus_queue.cc:237] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:12:41.054324 27741 tablet_service.cc:1514] Processing DeleteTablet for tablet fc9e0f35ebcd48b5a27c57b2485e599e with delete_type TABLET_DATA_TOMBSTONED (TS 6331f77dae9d452184e8cd0cbf5f1b43 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:41.055064 1169 tablet_replica.cc:330] T fc9e0f35ebcd48b5a27c57b2485e599e P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:41.055743 1169 raft_consensus.cc:2229] T fc9e0f35ebcd48b5a27c57b2485e599e P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:41.056382 1169 raft_consensus.cc:2258] T fc9e0f35ebcd48b5a27c57b2485e599e P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:41.057339 1169 ts_tablet_manager.cc:1875] T fc9e0f35ebcd48b5a27c57b2485e599e P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 08:12:41.060407 30872 consensus_peers.cc:485] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5d7223054aa4308917fd402d65ded5d. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:41.060500 30534 raft_consensus.cc:1261] T b5d7223054aa4308917fd402d65ded5d P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:41.062083 456 consensus_queue.cc:1035] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:41.063560 30672 raft_consensus.cc:1261] T b5d7223054aa4308917fd402d65ded5d P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:41.063920 28149 raft_consensus.cc:1261] T 8ab0410a98974002ab68dd0f6a456d59 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:12:41.065084 606 consensus_queue.cc:1035] T 8ab0410a98974002ab68dd0f6a456d59 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:41.065917 1169 ts_tablet_manager.cc:1888] T fc9e0f35ebcd48b5a27c57b2485e599e P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:41.067445 1169 log.cc:1196] T fc9e0f35ebcd48b5a27c57b2485e599e P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/fc9e0f35ebcd48b5a27c57b2485e599e
I20240501 08:12:41.068528 456 consensus_queue.cc:1035] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:41.076144 456 raft_consensus.cc:2932] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:41.078864 30672 raft_consensus.cc:2932] T b5d7223054aa4308917fd402d65ded5d P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:41.079236 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet fc9e0f35ebcd48b5a27c57b2485e599e (table rebalancer_extra_table_28 [id=474d31f683724897b682ce8741ddaa19]) successfully deleted
I20240501 08:12:41.087325 27615 catalog_manager.cc:5484] T b5d7223054aa4308917fd402d65ded5d P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:41.097378 30537 raft_consensus.cc:2932] T b5d7223054aa4308917fd402d65ded5d P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:41.117478 1137 tablet_bootstrap.cc:492] T 05b2485fa01a4706a85ebc150306637f P a29cd590999b4f32849ff5689cd53c11: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:41.118113 1137 tablet_bootstrap.cc:492] T 05b2485fa01a4706a85ebc150306637f P a29cd590999b4f32849ff5689cd53c11: Bootstrap complete.
I20240501 08:12:41.118404 1137 ts_tablet_manager.cc:1367] T 05b2485fa01a4706a85ebc150306637f P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.204s user 0.051s sys 0.012s
I20240501 08:12:41.119177 1137 raft_consensus.cc:348] T 05b2485fa01a4706a85ebc150306637f P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:12:41.119446 1137 raft_consensus.cc:729] T 05b2485fa01a4706a85ebc150306637f P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: LEARNER
I20240501 08:12:41.119681 1137 consensus_queue.cc:260] T 05b2485fa01a4706a85ebc150306637f P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:12:41.127877 1137 ts_tablet_manager.cc:1398] T 05b2485fa01a4706a85ebc150306637f P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 08:12:41.129804 28172 tablet_copy_service.cc:342] P 91d11eae5ef6432ca86eb8ed446b6ea6: Request end of tablet copy session a29cd590999b4f32849ff5689cd53c11-05b2485fa01a4706a85ebc150306637f received from {username='slave'} at 127.21.84.136:39335
I20240501 08:12:41.129981 28172 tablet_copy_service.cc:433] P 91d11eae5ef6432ca86eb8ed446b6ea6: ending tablet copy session a29cd590999b4f32849ff5689cd53c11-05b2485fa01a4706a85ebc150306637f on tablet 05b2485fa01a4706a85ebc150306637f with peer a29cd590999b4f32849ff5689cd53c11
I20240501 08:12:41.130367 30804 raft_consensus.cc:1261] T 8ab0410a98974002ab68dd0f6a456d59 P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:12:41.135075 1100 consensus_queue.cc:1035] T 8ab0410a98974002ab68dd0f6a456d59 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:41.155529 1123 ts_tablet_manager.cc:902] T 448a24d99bbe4fe4a571dc49a04ee275 P 31811708c6774037b82df44024b229b6: Initiating tablet copy from peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:41.165534 27760 raft_consensus.cc:1261] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:12:41.166589 456 consensus_queue.cc:1035] T 8ab0410a98974002ab68dd0f6a456d59 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:41.173475 28149 consensus_queue.cc:237] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:12:41.182551 1185 ts_tablet_manager.cc:902] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 91d11eae5ef6432ca86eb8ed446b6ea6: Initiating tablet copy from peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:41.193558 28018 raft_consensus.cc:1261] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:12:41.194195 28087 consensus_peers.cc:485] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 -> Peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): Couldn't send request to peer a29cd590999b4f32849ff5689cd53c11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b3e3d39f82549079293d98357884bf0. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:41.194368 32293 consensus_queue.cc:1035] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:41.195966 27890 raft_consensus.cc:1261] T 4b3e3d39f82549079293d98357884bf0 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:41.196628 32293 consensus_queue.cc:1035] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:41.218864 1185 tablet_copy_client.cc:308] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.136:46845
I20240501 08:12:41.219861 30957 tablet_copy_service.cc:140] P a29cd590999b4f32849ff5689cd53c11: Received BeginTabletCopySession request for tablet 8bbd50f618d14aad8e45c2aa6fbabc1f from peer 91d11eae5ef6432ca86eb8ed446b6ea6 ({username='slave'} at 127.21.84.132:58509)
I20240501 08:12:41.220075 30957 tablet_copy_service.cc:161] P a29cd590999b4f32849ff5689cd53c11: Beginning new tablet copy session on tablet 8bbd50f618d14aad8e45c2aa6fbabc1f from peer 91d11eae5ef6432ca86eb8ed446b6ea6 at {username='slave'} at 127.21.84.132:58509: session id = 91d11eae5ef6432ca86eb8ed446b6ea6-8bbd50f618d14aad8e45c2aa6fbabc1f
I20240501 08:12:41.225001 30957 tablet_copy_source_session.cc:205] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:41.225734 307 raft_consensus.cc:2932] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:41.238582 28018 raft_consensus.cc:2932] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:41.242115 1185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bbd50f618d14aad8e45c2aa6fbabc1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:41.248314 1185 tablet_copy_client.cc:783] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:41.248678 1185 tablet_copy_client.cc:647] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:41.250705 1185 tablet_copy_client.cc:523] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:41.255363 1185 tablet_bootstrap.cc:492] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:41.254498 27615 catalog_manager.cc:5484] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: config changed from index -1 to 4, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:41.263732 30934 raft_consensus.cc:1203] T 05b2485fa01a4706a85ebc150306637f P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:41.267951 1185 tablet_bootstrap.cc:492] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:41.270282 1185 tablet_bootstrap.cc:492] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap complete.
I20240501 08:12:41.270998 1185 ts_tablet_manager.cc:1367] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.001s
I20240501 08:12:41.272260 1185 raft_consensus.cc:348] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:12:41.274003 1185 raft_consensus.cc:729] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: LEARNER
I20240501 08:12:41.274399 1185 consensus_queue.cc:260] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:12:41.275396 1185 ts_tablet_manager.cc:1398] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 08:12:41.275614 30536 raft_consensus.cc:1203] T 6c04880ea87d4894bd3a63d94f4a69f4 P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:41.277156 1123 tablet_copy_client.cc:308] T 448a24d99bbe4fe4a571dc49a04ee275 P 31811708c6774037b82df44024b229b6: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.134:39933
I20240501 08:12:41.278234 30693 tablet_copy_service.cc:140] P 9a022e1a501f4d44a0468054d6c3de47: Received BeginTabletCopySession request for tablet 448a24d99bbe4fe4a571dc49a04ee275 from peer 31811708c6774037b82df44024b229b6 ({username='slave'} at 127.21.84.133:36309)
I20240501 08:12:41.278467 30693 tablet_copy_service.cc:161] P 9a022e1a501f4d44a0468054d6c3de47: Beginning new tablet copy session on tablet 448a24d99bbe4fe4a571dc49a04ee275 from peer 31811708c6774037b82df44024b229b6 at {username='slave'} at 127.21.84.133:36309: session id = 31811708c6774037b82df44024b229b6-448a24d99bbe4fe4a571dc49a04ee275
I20240501 08:12:41.279749 30957 tablet_copy_service.cc:342] P a29cd590999b4f32849ff5689cd53c11: Request end of tablet copy session 91d11eae5ef6432ca86eb8ed446b6ea6-8bbd50f618d14aad8e45c2aa6fbabc1f received from {username='slave'} at 127.21.84.132:58509
I20240501 08:12:41.279898 30957 tablet_copy_service.cc:433] P a29cd590999b4f32849ff5689cd53c11: ending tablet copy session 91d11eae5ef6432ca86eb8ed446b6ea6-8bbd50f618d14aad8e45c2aa6fbabc1f on tablet 8bbd50f618d14aad8e45c2aa6fbabc1f with peer 91d11eae5ef6432ca86eb8ed446b6ea6
I20240501 08:12:41.281890 30693 tablet_copy_source_session.cc:205] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:41.284409 1137 ts_tablet_manager.cc:902] T d020e1225d5d447f993f920569a05f3d P a29cd590999b4f32849ff5689cd53c11: Initiating tablet copy from peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:41.286471 1123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 448a24d99bbe4fe4a571dc49a04ee275. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:41.293105 30803 raft_consensus.cc:1203] T 0be6dad1a409426c9a3dc0866505b3ea P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:41.293684 1123 tablet_copy_client.cc:783] T 448a24d99bbe4fe4a571dc49a04ee275 P 31811708c6774037b82df44024b229b6: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:41.294090 1123 tablet_copy_client.cc:647] T 448a24d99bbe4fe4a571dc49a04ee275 P 31811708c6774037b82df44024b229b6: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:41.301374 30934 consensus_queue.cc:237] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:12:41.304754 1123 tablet_copy_client.cc:523] T 448a24d99bbe4fe4a571dc49a04ee275 P 31811708c6774037b82df44024b229b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:41.310487 1123 tablet_bootstrap.cc:492] T 448a24d99bbe4fe4a571dc49a04ee275 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:41.315080 30935 consensus_queue.cc:237] T 8ab0410a98974002ab68dd0f6a456d59 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
W20240501 08:12:41.317080 28088 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.130:36503, user_credentials={real_user=slave}} blocked reactor thread for 70167.7us
I20240501 08:12:41.318701 30803 raft_consensus.cc:1261] T 8ab0410a98974002ab68dd0f6a456d59 P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:12:41.319778 1100 consensus_queue.cc:1035] T 8ab0410a98974002ab68dd0f6a456d59 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:41.323287 28149 raft_consensus.cc:1261] T 8ab0410a98974002ab68dd0f6a456d59 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:12:41.324144 456 raft_consensus.cc:2932] T 8ab0410a98974002ab68dd0f6a456d59 P a29cd590999b4f32849ff5689cd53c11 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:41.324090 27890 raft_consensus.cc:2932] T 4b3e3d39f82549079293d98357884bf0 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:41.324864 30803 raft_consensus.cc:2932] T 8ab0410a98974002ab68dd0f6a456d59 P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:41.326583 1100 consensus_queue.cc:1035] T 8ab0410a98974002ab68dd0f6a456d59 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:41.328445 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8ab0410a98974002ab68dd0f6a456d59 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:41.332147 27619 catalog_manager.cc:5484] T 8ab0410a98974002ab68dd0f6a456d59 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index 3 to 5, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New cstate: current_term: 2 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:41.337533 27741 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ab0410a98974002ab68dd0f6a456d59 with delete_type TABLET_DATA_TOMBSTONED (TS 6331f77dae9d452184e8cd0cbf5f1b43 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:41.337905 1169 tablet_replica.cc:330] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:41.338127 1169 raft_consensus.cc:2229] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:41.338306 1169 raft_consensus.cc:2258] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:41.339430 1169 ts_tablet_manager.cc:1875] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:41.345242 1123 tablet_bootstrap.cc:492] T 448a24d99bbe4fe4a571dc49a04ee275 P 31811708c6774037b82df44024b229b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:41.345692 1123 tablet_bootstrap.cc:492] T 448a24d99bbe4fe4a571dc49a04ee275 P 31811708c6774037b82df44024b229b6: Bootstrap complete.
I20240501 08:12:41.345988 1123 ts_tablet_manager.cc:1367] T 448a24d99bbe4fe4a571dc49a04ee275 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.036s user 0.006s sys 0.002s
I20240501 08:12:41.347188 1123 raft_consensus.cc:348] T 448a24d99bbe4fe4a571dc49a04ee275 P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
I20240501 08:12:41.347537 1123 raft_consensus.cc:729] T 448a24d99bbe4fe4a571dc49a04ee275 P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: LEARNER
I20240501 08:12:41.347805 1123 consensus_queue.cc:260] T 448a24d99bbe4fe4a571dc49a04ee275 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
I20240501 08:12:41.349102 27892 raft_consensus.cc:1261] T 87804550560d45269e6d4d6fe8938f55 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:41.349436 30804 raft_consensus.cc:1261] T 87804550560d45269e6d4d6fe8938f55 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:41.350000 456 consensus_queue.cc:1035] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:41.350706 456 consensus_queue.cc:1035] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:41.352692 1212 ts_tablet_manager.cc:902] T 2e1dd7a48a574ffcbea936af5b0fe192 P 9a022e1a501f4d44a0468054d6c3de47: Initiating tablet copy from peer 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:41.355494 1123 ts_tablet_manager.cc:1398] T 448a24d99bbe4fe4a571dc49a04ee275 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
W20240501 08:12:41.360972 30872 consensus_peers.cc:485] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 -> Peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Couldn't send request to peer aad81047009a4ffcbc450b60030583b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87804550560d45269e6d4d6fe8938f55. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:41.362501 1137 tablet_copy_client.cc:308] T d020e1225d5d447f993f920569a05f3d P a29cd590999b4f32849ff5689cd53c11: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.133:38293
I20240501 08:12:41.364504 28151 raft_consensus.cc:2932] T 8ab0410a98974002ab68dd0f6a456d59 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:41.365407 30539 tablet_copy_service.cc:140] P 31811708c6774037b82df44024b229b6: Received BeginTabletCopySession request for tablet d020e1225d5d447f993f920569a05f3d from peer a29cd590999b4f32849ff5689cd53c11 ({username='slave'} at 127.21.84.136:55871)
I20240501 08:12:41.365598 30539 tablet_copy_service.cc:161] P 31811708c6774037b82df44024b229b6: Beginning new tablet copy session on tablet d020e1225d5d447f993f920569a05f3d from peer a29cd590999b4f32849ff5689cd53c11 at {username='slave'} at 127.21.84.136:55871: session id = a29cd590999b4f32849ff5689cd53c11-d020e1225d5d447f993f920569a05f3d
I20240501 08:12:41.366354 456 raft_consensus.cc:2932] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:41.367055 27892 raft_consensus.cc:2932] T 87804550560d45269e6d4d6fe8938f55 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:41.368638 30539 tablet_copy_source_session.cc:205] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:41.372623 30693 tablet_copy_service.cc:342] P 9a022e1a501f4d44a0468054d6c3de47: Request end of tablet copy session 31811708c6774037b82df44024b229b6-448a24d99bbe4fe4a571dc49a04ee275 received from {username='slave'} at 127.21.84.133:36309
I20240501 08:12:41.372782 30693 tablet_copy_service.cc:433] P 9a022e1a501f4d44a0468054d6c3de47: ending tablet copy session 31811708c6774037b82df44024b229b6-448a24d99bbe4fe4a571dc49a04ee275 on tablet 448a24d99bbe4fe4a571dc49a04ee275 with peer 31811708c6774037b82df44024b229b6
I20240501 08:12:41.374070 27612 catalog_manager.cc:5484] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:41.378952 30804 raft_consensus.cc:2932] T 87804550560d45269e6d4d6fe8938f55 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:41.405472 1137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d020e1225d5d447f993f920569a05f3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:41.407137 1169 ts_tablet_manager.cc:1888] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240501 08:12:41.407493 1132 ts_tablet_manager.cc:902] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5: Initiating tablet copy from peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:41.407946 1169 log.cc:1196] T 8ab0410a98974002ab68dd0f6a456d59 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/8ab0410a98974002ab68dd0f6a456d59
I20240501 08:12:41.415661 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet 8ab0410a98974002ab68dd0f6a456d59 (table rebalancer_extra_table_10 [id=6377dbf65f184f9cbe44f12bd6a0405f]) successfully deleted
I20240501 08:12:41.437064 1212 tablet_copy_client.cc:308] T 2e1dd7a48a574ffcbea936af5b0fe192 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.135:33503
I20240501 08:12:41.437970 30825 tablet_copy_service.cc:140] P 64998323a1f4482d8704bf20594d13a5: Received BeginTabletCopySession request for tablet 2e1dd7a48a574ffcbea936af5b0fe192 from peer 9a022e1a501f4d44a0468054d6c3de47 ({username='slave'} at 127.21.84.134:53241)
I20240501 08:12:41.438189 30825 tablet_copy_service.cc:161] P 64998323a1f4482d8704bf20594d13a5: Beginning new tablet copy session on tablet 2e1dd7a48a574ffcbea936af5b0fe192 from peer 9a022e1a501f4d44a0468054d6c3de47 at {username='slave'} at 127.21.84.134:53241: session id = 9a022e1a501f4d44a0468054d6c3de47-2e1dd7a48a574ffcbea936af5b0fe192
I20240501 08:12:41.438716 1137 tablet_copy_client.cc:783] T d020e1225d5d447f993f920569a05f3d P a29cd590999b4f32849ff5689cd53c11: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:41.438968 1137 tablet_copy_client.cc:647] T d020e1225d5d447f993f920569a05f3d P a29cd590999b4f32849ff5689cd53c11: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:41.441063 30825 tablet_copy_source_session.cc:205] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:41.442834 1212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e1dd7a48a574ffcbea936af5b0fe192. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:41.449368 1137 tablet_copy_client.cc:523] T d020e1225d5d447f993f920569a05f3d P a29cd590999b4f32849ff5689cd53c11: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:41.454533 1137 tablet_bootstrap.cc:492] T d020e1225d5d447f993f920569a05f3d P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:41.477205 1212 tablet_copy_client.cc:783] T 2e1dd7a48a574ffcbea936af5b0fe192 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:41.477574 1212 tablet_copy_client.cc:647] T 2e1dd7a48a574ffcbea936af5b0fe192 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:41.479887 1137 tablet_bootstrap.cc:492] T d020e1225d5d447f993f920569a05f3d P a29cd590999b4f32849ff5689cd53c11: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:41.479864 1212 tablet_copy_client.cc:523] T 2e1dd7a48a574ffcbea936af5b0fe192 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:41.480346 1137 tablet_bootstrap.cc:492] T d020e1225d5d447f993f920569a05f3d P a29cd590999b4f32849ff5689cd53c11: Bootstrap complete.
I20240501 08:12:41.480618 1137 ts_tablet_manager.cc:1367] T d020e1225d5d447f993f920569a05f3d P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.001s
I20240501 08:12:41.481370 1137 raft_consensus.cc:348] T d020e1225d5d447f993f920569a05f3d P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:12:41.481632 1137 raft_consensus.cc:729] T d020e1225d5d447f993f920569a05f3d P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: LEARNER
I20240501 08:12:41.481843 1137 consensus_queue.cc:260] T d020e1225d5d447f993f920569a05f3d P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:12:41.482434 1137 ts_tablet_manager.cc:1398] T d020e1225d5d447f993f920569a05f3d P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:41.483412 1212 tablet_bootstrap.cc:492] T 2e1dd7a48a574ffcbea936af5b0fe192 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:41.484705 30539 tablet_copy_service.cc:342] P 31811708c6774037b82df44024b229b6: Request end of tablet copy session a29cd590999b4f32849ff5689cd53c11-d020e1225d5d447f993f920569a05f3d received from {username='slave'} at 127.21.84.136:55871
I20240501 08:12:41.484822 30539 tablet_copy_service.cc:433] P 31811708c6774037b82df44024b229b6: ending tablet copy session a29cd590999b4f32849ff5689cd53c11-d020e1225d5d447f993f920569a05f3d on tablet d020e1225d5d447f993f920569a05f3d with peer a29cd590999b4f32849ff5689cd53c11
I20240501 08:12:41.497073 1132 tablet_copy_client.cc:308] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.129:43471
I20240501 08:12:41.497943 27781 tablet_copy_service.cc:140] P 6331f77dae9d452184e8cd0cbf5f1b43: Received BeginTabletCopySession request for tablet 32a54b70dcc447828ec293a9ff0afb80 from peer 64998323a1f4482d8704bf20594d13a5 ({username='slave'} at 127.21.84.135:52115)
I20240501 08:12:41.498098 27781 tablet_copy_service.cc:161] P 6331f77dae9d452184e8cd0cbf5f1b43: Beginning new tablet copy session on tablet 32a54b70dcc447828ec293a9ff0afb80 from peer 64998323a1f4482d8704bf20594d13a5 at {username='slave'} at 127.21.84.135:52115: session id = 64998323a1f4482d8704bf20594d13a5-32a54b70dcc447828ec293a9ff0afb80
I20240501 08:12:41.500919 27781 tablet_copy_source_session.cc:205] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:41.501348 1212 tablet_bootstrap.cc:492] T 2e1dd7a48a574ffcbea936af5b0fe192 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:41.501902 1212 tablet_bootstrap.cc:492] T 2e1dd7a48a574ffcbea936af5b0fe192 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap complete.
I20240501 08:12:41.502199 1212 ts_tablet_manager.cc:1367] T 2e1dd7a48a574ffcbea936af5b0fe192 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.007s
I20240501 08:12:41.503062 1212 raft_consensus.cc:348] T 2e1dd7a48a574ffcbea936af5b0fe192 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:12:41.503304 1212 raft_consensus.cc:729] T 2e1dd7a48a574ffcbea936af5b0fe192 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: LEARNER
I20240501 08:12:41.504351 1212 consensus_queue.cc:260] T 2e1dd7a48a574ffcbea936af5b0fe192 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:12:41.506557 1212 ts_tablet_manager.cc:1398] T 2e1dd7a48a574ffcbea936af5b0fe192 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 08:12:41.507795 30825 tablet_copy_service.cc:342] P 64998323a1f4482d8704bf20594d13a5: Request end of tablet copy session 9a022e1a501f4d44a0468054d6c3de47-2e1dd7a48a574ffcbea936af5b0fe192 received from {username='slave'} at 127.21.84.134:53241
I20240501 08:12:41.507962 30825 tablet_copy_service.cc:433] P 64998323a1f4482d8704bf20594d13a5: ending tablet copy session 9a022e1a501f4d44a0468054d6c3de47-2e1dd7a48a574ffcbea936af5b0fe192 on tablet 2e1dd7a48a574ffcbea936af5b0fe192 with peer 9a022e1a501f4d44a0468054d6c3de47
I20240501 08:12:41.514647 1132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a54b70dcc447828ec293a9ff0afb80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:41.535898 27760 consensus_queue.cc:237] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
I20240501 08:12:41.552095 1132 tablet_copy_client.cc:783] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:41.552372 1132 tablet_copy_client.cc:647] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:41.562570 30536 raft_consensus.cc:1203] T 448a24d99bbe4fe4a571dc49a04ee275 P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:41.563875 1132 tablet_copy_client.cc:523] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:41.570732 1132 tablet_bootstrap.cc:492] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:41.574394 28018 raft_consensus.cc:1261] T bb7dae019c7e4842b0daa35ee4b40367 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:41.574810 28151 raft_consensus.cc:1261] T bb7dae019c7e4842b0daa35ee4b40367 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:41.575069 1087 consensus_queue.cc:1035] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:41.576741 1089 consensus_queue.cc:1035] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:12:41.593703 27697 consensus_peers.cc:485] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43 -> Peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): Couldn't send request to peer 31811708c6774037b82df44024b229b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb7dae019c7e4842b0daa35ee4b40367. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:41.607520 1089 raft_consensus.cc:2932] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:12:41.611196 28149 raft_consensus.cc:2932] T bb7dae019c7e4842b0daa35ee4b40367 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:12:41.624239 27615 catalog_manager.cc:5484] T bb7dae019c7e4842b0daa35ee4b40367 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: config changed from index -1 to 4, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:12:41.627800 28018 raft_consensus.cc:2932] T bb7dae019c7e4842b0daa35ee4b40367 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:12:41.653380 1132 tablet_bootstrap.cc:492] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:41.653903 1132 tablet_bootstrap.cc:492] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5: Bootstrap complete.
I20240501 08:12:41.654202 1132 ts_tablet_manager.cc:1367] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.084s user 0.049s sys 0.007s
I20240501 08:12:41.655081 1132 raft_consensus.cc:348] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
I20240501 08:12:41.655369 1132 raft_consensus.cc:729] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: LEARNER
I20240501 08:12:41.655634 1132 consensus_queue.cc:260] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
I20240501 08:12:41.657562 1132 ts_tablet_manager.cc:1398] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:12:41.660760 27781 tablet_copy_service.cc:342] P 6331f77dae9d452184e8cd0cbf5f1b43: Request end of tablet copy session 64998323a1f4482d8704bf20594d13a5-32a54b70dcc447828ec293a9ff0afb80 received from {username='slave'} at 127.21.84.135:52115
I20240501 08:12:41.660923 27781 tablet_copy_service.cc:433] P 6331f77dae9d452184e8cd0cbf5f1b43: ending tablet copy session 64998323a1f4482d8704bf20594d13a5-32a54b70dcc447828ec293a9ff0afb80 on tablet 32a54b70dcc447828ec293a9ff0afb80 with peer 64998323a1f4482d8704bf20594d13a5
I20240501 08:12:41.671033 879 raft_consensus.cc:1050] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5: attempting to promote NON_VOTER 31811708c6774037b82df44024b229b6 to VOTER
I20240501 08:12:41.671882 879 consensus_queue.cc:237] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } }
I20240501 08:12:41.674048 28018 consensus_queue.cc:237] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:12:41.675693 27760 raft_consensus.cc:1261] T 6c04880ea87d4894bd3a63d94f4a69f4 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:41.675887 30536 raft_consensus.cc:1261] T 6c04880ea87d4894bd3a63d94f4a69f4 P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:41.676344 877 consensus_queue.cc:1035] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:41.676734 1246 consensus_queue.cc:1035] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:41.678017 30934 raft_consensus.cc:1261] T 95cb5060152e4237988a4c5723a62601 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:12:41.678637 27957 consensus_peers.cc:485] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 95cb5060152e4237988a4c5723a62601. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:41.679141 877 consensus_queue.cc:1035] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:41.691866 1137 ts_tablet_manager.cc:902] T 4b3e3d39f82549079293d98357884bf0 P a29cd590999b4f32849ff5689cd53c11: Initiating tablet copy from peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:41.692639 1251 ts_tablet_manager.cc:902] T b5d7223054aa4308917fd402d65ded5d P 6331f77dae9d452184e8cd0cbf5f1b43: Initiating tablet copy from peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:41.693010 28151 raft_consensus.cc:1261] T 6c04880ea87d4894bd3a63d94f4a69f4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:41.693564 1244 consensus_queue.cc:1035] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:41.694700 1244 raft_consensus.cc:2932] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:41.695451 30935 raft_consensus.cc:2932] T 95cb5060152e4237988a4c5723a62601 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:41.711661 27615 catalog_manager.cc:5484] T 95cb5060152e4237988a4c5723a62601 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:41.720192 877 raft_consensus.cc:2932] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:41.727313 30672 raft_consensus.cc:1261] T 95cb5060152e4237988a4c5723a62601 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:41.731251 28149 raft_consensus.cc:2932] T 6c04880ea87d4894bd3a63d94f4a69f4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:41.733013 28151 raft_consensus.cc:1203] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:41.733912 1244 consensus_queue.cc:1035] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:41.741654 30534 raft_consensus.cc:2932] T 6c04880ea87d4894bd3a63d94f4a69f4 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:41.742036 27760 raft_consensus.cc:2932] T 6c04880ea87d4894bd3a63d94f4a69f4 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:41.753393 27612 catalog_manager.cc:5484] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 4 to 5, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:41.762935 28020 consensus_queue.cc:237] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } }
I20240501 08:12:41.765288 30534 raft_consensus.cc:1261] T 6c04880ea87d4894bd3a63d94f4a69f4 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:41.765952 1246 consensus_queue.cc:1035] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:41.766945 27760 raft_consensus.cc:1261] T 6c04880ea87d4894bd3a63d94f4a69f4 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:41.768152 1246 consensus_queue.cc:1035] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:41.769693 30935 raft_consensus.cc:1203] T d020e1225d5d447f993f920569a05f3d P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:41.771481 1081 raft_consensus.cc:1050] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af: attempting to promote NON_VOTER 64998323a1f4482d8704bf20594d13a5 to VOTER
I20240501 08:12:41.772303 1081 consensus_queue.cc:237] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:12:41.773056 1251 tablet_copy_client.cc:308] T b5d7223054aa4308917fd402d65ded5d P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.136:46845
I20240501 08:12:41.773950 30957 tablet_copy_service.cc:140] P a29cd590999b4f32849ff5689cd53c11: Received BeginTabletCopySession request for tablet b5d7223054aa4308917fd402d65ded5d from peer 6331f77dae9d452184e8cd0cbf5f1b43 ({username='slave'} at 127.21.84.129:48739)
I20240501 08:12:41.774147 30957 tablet_copy_service.cc:161] P a29cd590999b4f32849ff5689cd53c11: Beginning new tablet copy session on tablet b5d7223054aa4308917fd402d65ded5d from peer 6331f77dae9d452184e8cd0cbf5f1b43 at {username='slave'} at 127.21.84.129:48739: session id = 6331f77dae9d452184e8cd0cbf5f1b43-b5d7223054aa4308917fd402d65ded5d
I20240501 08:12:41.777256 1246 raft_consensus.cc:2932] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:41.777742 27760 raft_consensus.cc:2932] T 6c04880ea87d4894bd3a63d94f4a69f4 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:41.778084 30957 tablet_copy_source_session.cc:205] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:41.784173 30534 raft_consensus.cc:2932] T 6c04880ea87d4894bd3a63d94f4a69f4 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:41.785187 1251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5d7223054aa4308917fd402d65ded5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:41.787477 27611 catalog_manager.cc:5484] T 6c04880ea87d4894bd3a63d94f4a69f4 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index 5 to 6, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:41.787824 307 raft_consensus.cc:1050] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6: attempting to promote NON_VOTER a29cd590999b4f32849ff5689cd53c11 to VOTER
I20240501 08:12:41.788607 307 consensus_queue.cc:237] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:12:41.790887 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6c04880ea87d4894bd3a63d94f4a69f4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:41.792264 28131 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c04880ea87d4894bd3a63d94f4a69f4 with delete_type TABLET_DATA_TOMBSTONED (TS 91d11eae5ef6432ca86eb8ed446b6ea6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:41.793402 30804 raft_consensus.cc:1261] T 0be6dad1a409426c9a3dc0866505b3ea P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:41.793779 28020 raft_consensus.cc:1261] T 0be6dad1a409426c9a3dc0866505b3ea P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:41.794140 810 consensus_queue.cc:1035] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:41.797722 810 consensus_queue.cc:1035] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:41.801460 30935 raft_consensus.cc:1261] T 05b2485fa01a4706a85ebc150306637f P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:41.802312 840 consensus_queue.cc:1035] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:41.803920 28019 raft_consensus.cc:1261] T 05b2485fa01a4706a85ebc150306637f P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:41.804071 27760 raft_consensus.cc:1261] T 0be6dad1a409426c9a3dc0866505b3ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:41.804756 810 consensus_queue.cc:1035] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:41.805927 840 consensus_queue.cc:1035] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:41.807529 1251 tablet_copy_client.cc:783] T b5d7223054aa4308917fd402d65ded5d P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:41.808701 1251 tablet_copy_client.cc:647] T b5d7223054aa4308917fd402d65ded5d P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:41.823289 27890 raft_consensus.cc:1261] T 05b2485fa01a4706a85ebc150306637f P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:41.824112 840 consensus_queue.cc:1035] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:41.833659 30672 raft_consensus.cc:2932] T 95cb5060152e4237988a4c5723a62601 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:41.836927 1261 ts_tablet_manager.cc:902] T 87804550560d45269e6d4d6fe8938f55 P aad81047009a4ffcbc450b60030583b5: Initiating tablet copy from peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:41.838387 810 raft_consensus.cc:2932] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:41.845980 28020 raft_consensus.cc:2932] T 0be6dad1a409426c9a3dc0866505b3ea P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:41.846186 30804 raft_consensus.cc:2932] T 0be6dad1a409426c9a3dc0866505b3ea P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:41.866845 1269 tablet_replica.cc:330] T 6c04880ea87d4894bd3a63d94f4a69f4 P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:41.868012 27611 catalog_manager.cc:5484] T 0be6dad1a409426c9a3dc0866505b3ea P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:41.869560 1137 tablet_copy_client.cc:308] T 4b3e3d39f82549079293d98357884bf0 P a29cd590999b4f32849ff5689cd53c11: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.132:41323
I20240501 08:12:41.872907 1251 tablet_copy_client.cc:523] T b5d7223054aa4308917fd402d65ded5d P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:41.875192 27760 raft_consensus.cc:2932] T 0be6dad1a409426c9a3dc0866505b3ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:41.870381 28172 tablet_copy_service.cc:140] P 91d11eae5ef6432ca86eb8ed446b6ea6: Received BeginTabletCopySession request for tablet 4b3e3d39f82549079293d98357884bf0 from peer a29cd590999b4f32849ff5689cd53c11 ({username='slave'} at 127.21.84.136:39335)
I20240501 08:12:41.877570 28172 tablet_copy_service.cc:161] P 91d11eae5ef6432ca86eb8ed446b6ea6: Beginning new tablet copy session on tablet 4b3e3d39f82549079293d98357884bf0 from peer a29cd590999b4f32849ff5689cd53c11 at {username='slave'} at 127.21.84.136:39335: session id = a29cd590999b4f32849ff5689cd53c11-4b3e3d39f82549079293d98357884bf0
I20240501 08:12:41.878922 1251 tablet_bootstrap.cc:492] T b5d7223054aa4308917fd402d65ded5d P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:41.880283 28172 tablet_copy_source_session.cc:205] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:41.880911 1269 raft_consensus.cc:2229] T 6c04880ea87d4894bd3a63d94f4a69f4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:41.881145 1269 raft_consensus.cc:2258] T 6c04880ea87d4894bd3a63d94f4a69f4 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:41.885561 1137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b3e3d39f82549079293d98357884bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:41.886823 840 raft_consensus.cc:2932] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:41.885406 27890 consensus_queue.cc:237] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:12:41.890704 30934 raft_consensus.cc:2932] T 05b2485fa01a4706a85ebc150306637f P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:41.900213 27891 raft_consensus.cc:2932] T 05b2485fa01a4706a85ebc150306637f P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:41.901234 27760 raft_consensus.cc:1261] T 0be6dad1a409426c9a3dc0866505b3ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:41.901139 27615 catalog_manager.cc:5484] T 05b2485fa01a4706a85ebc150306637f P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:41.901891 1279 consensus_queue.cc:1035] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:41.914398 30802 raft_consensus.cc:1203] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:41.915179 30803 raft_consensus.cc:1261] T 0be6dad1a409426c9a3dc0866505b3ea P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:41.919597 1081 consensus_queue.cc:1035] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:41.920679 1251 tablet_bootstrap.cc:492] T b5d7223054aa4308917fd402d65ded5d P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:41.924993 1278 raft_consensus.cc:2932] T 0be6dad1a409426c9a3dc0866505b3ea P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:41.926282 30800 raft_consensus.cc:2932] T 0be6dad1a409426c9a3dc0866505b3ea P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:41.929490 1251 tablet_bootstrap.cc:492] T b5d7223054aa4308917fd402d65ded5d P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap complete.
I20240501 08:12:41.930560 1251 ts_tablet_manager.cc:1367] T b5d7223054aa4308917fd402d65ded5d P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.052s user 0.006s sys 0.002s
I20240501 08:12:41.931593 1251 raft_consensus.cc:348] T b5d7223054aa4308917fd402d65ded5d P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:12:41.934577 1269 ts_tablet_manager.cc:1875] T 6c04880ea87d4894bd3a63d94f4a69f4 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:41.934718 28018 raft_consensus.cc:2932] T 05b2485fa01a4706a85ebc150306637f P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:41.940234 27607 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0be6dad1a409426c9a3dc0866505b3ea with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:41.940202 27758 raft_consensus.cc:2932] T 0be6dad1a409426c9a3dc0866505b3ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:41.947486 27619 catalog_manager.cc:5484] T 0be6dad1a409426c9a3dc0866505b3ea P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:41.940186 1251 raft_consensus.cc:729] T b5d7223054aa4308917fd402d65ded5d P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: LEARNER
I20240501 08:12:41.948717 1251 consensus_queue.cc:260] T b5d7223054aa4308917fd402d65ded5d P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:12:41.961079 28000 tablet_service.cc:1514] Processing DeleteTablet for tablet 0be6dad1a409426c9a3dc0866505b3ea with delete_type TABLET_DATA_TOMBSTONED (TS aad81047009a4ffcbc450b60030583b5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:41.963373 30669 raft_consensus.cc:1203] T 2e1dd7a48a574ffcbea936af5b0fe192 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:41.963456 28149 consensus_queue.cc:237] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:12:41.966588 1251 ts_tablet_manager.cc:1398] T b5d7223054aa4308917fd402d65ded5d P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.036s user 0.002s sys 0.000s
I20240501 08:12:41.968679 1286 ts_tablet_manager.cc:902] T bb7dae019c7e4842b0daa35ee4b40367 P 31811708c6774037b82df44024b229b6: Initiating tablet copy from peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:41.969065 28020 raft_consensus.cc:1261] T 05b2485fa01a4706a85ebc150306637f P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:41.970525 840 consensus_queue.cc:1035] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:41.970599 30935 raft_consensus.cc:1261] T 05b2485fa01a4706a85ebc150306637f P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:41.971346 840 consensus_queue.cc:1035] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:41.972024 1137 tablet_copy_client.cc:783] T 4b3e3d39f82549079293d98357884bf0 P a29cd590999b4f32849ff5689cd53c11: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:41.972261 1137 tablet_copy_client.cc:647] T 4b3e3d39f82549079293d98357884bf0 P a29cd590999b4f32849ff5689cd53c11: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:41.972859 30534 consensus_queue.cc:237] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:41.975029 30957 tablet_copy_service.cc:342] P a29cd590999b4f32849ff5689cd53c11: Request end of tablet copy session 6331f77dae9d452184e8cd0cbf5f1b43-b5d7223054aa4308917fd402d65ded5d received from {username='slave'} at 127.21.84.129:48739
I20240501 08:12:41.975175 30957 tablet_copy_service.cc:433] P a29cd590999b4f32849ff5689cd53c11: ending tablet copy session 6331f77dae9d452184e8cd0cbf5f1b43-b5d7223054aa4308917fd402d65ded5d on tablet b5d7223054aa4308917fd402d65ded5d with peer 6331f77dae9d452184e8cd0cbf5f1b43
I20240501 08:12:41.976289 840 raft_consensus.cc:2932] T 05b2485fa01a4706a85ebc150306637f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:41.979552 1269 ts_tablet_manager.cc:1888] T 6c04880ea87d4894bd3a63d94f4a69f4 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:41.979760 1269 log.cc:1196] T 6c04880ea87d4894bd3a63d94f4a69f4 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/6c04880ea87d4894bd3a63d94f4a69f4
I20240501 08:12:41.981246 27604 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 05b2485fa01a4706a85ebc150306637f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:41.981488 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet 6c04880ea87d4894bd3a63d94f4a69f4 (table rebalance_test_table_7 [id=1ae463b433bd4ed6bb9ab1bdc10dfdf6]) successfully deleted
I20240501 08:12:41.982697 1137 tablet_copy_client.cc:523] T 4b3e3d39f82549079293d98357884bf0 P a29cd590999b4f32849ff5689cd53c11: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:41.984333 30935 raft_consensus.cc:2932] T 05b2485fa01a4706a85ebc150306637f P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
W20240501 08:12:41.989271 30473 consensus_peers.cc:485] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f8c0a8e86f04f32b79743f819948544. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:41.991078 28019 raft_consensus.cc:1261] T 4f8c0a8e86f04f32b79743f819948544 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:41.991266 30934 raft_consensus.cc:1261] T 4f8c0a8e86f04f32b79743f819948544 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:41.991932 523 consensus_queue.cc:1035] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:41.992509 939 consensus_queue.cc:1035] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:42.002364 1137 tablet_bootstrap.cc:492] T 4b3e3d39f82549079293d98357884bf0 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:42.003947 27619 catalog_manager.cc:5484] T 05b2485fa01a4706a85ebc150306637f P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index 5 to 6, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:42.007095 28022 raft_consensus.cc:2932] T 05b2485fa01a4706a85ebc150306637f P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:42.009085 523 raft_consensus.cc:2932] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:42.009825 28020 raft_consensus.cc:2932] T 4f8c0a8e86f04f32b79743f819948544 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:42.017077 27871 tablet_service.cc:1514] Processing DeleteTablet for tablet 05b2485fa01a4706a85ebc150306637f with delete_type TABLET_DATA_TOMBSTONED (TS f1525cbc82a24d13b65f136695c903af not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:42.018064 27618 catalog_manager.cc:5484] T 4f8c0a8e86f04f32b79743f819948544 P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:42.031304 1287 tablet_replica.cc:330] T 0be6dad1a409426c9a3dc0866505b3ea P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:42.031795 1287 raft_consensus.cc:2229] T 0be6dad1a409426c9a3dc0866505b3ea P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:42.034890 1287 raft_consensus.cc:2258] T 0be6dad1a409426c9a3dc0866505b3ea P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:42.035079 1261 tablet_copy_client.cc:308] T 87804550560d45269e6d4d6fe8938f55 P aad81047009a4ffcbc450b60030583b5: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.136:46845
I20240501 08:12:42.036392 30957 tablet_copy_service.cc:140] P a29cd590999b4f32849ff5689cd53c11: Received BeginTabletCopySession request for tablet 87804550560d45269e6d4d6fe8938f55 from peer aad81047009a4ffcbc450b60030583b5 ({username='slave'} at 127.21.84.131:60955)
I20240501 08:12:42.036581 30957 tablet_copy_service.cc:161] P a29cd590999b4f32849ff5689cd53c11: Beginning new tablet copy session on tablet 87804550560d45269e6d4d6fe8938f55 from peer aad81047009a4ffcbc450b60030583b5 at {username='slave'} at 127.21.84.131:60955: session id = aad81047009a4ffcbc450b60030583b5-87804550560d45269e6d4d6fe8938f55
I20240501 08:12:42.038486 1287 ts_tablet_manager.cc:1875] T 0be6dad1a409426c9a3dc0866505b3ea P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:42.039554 30957 tablet_copy_source_session.cc:205] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:42.043258 30934 raft_consensus.cc:2932] T 4f8c0a8e86f04f32b79743f819948544 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:42.059474 1233 raft_consensus.cc:1050] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47: attempting to promote NON_VOTER 31811708c6774037b82df44024b229b6 to VOTER
I20240501 08:12:42.060355 1233 consensus_queue.cc:237] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } }
I20240501 08:12:42.060626 1261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87804550560d45269e6d4d6fe8938f55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:42.084949 1292 tablet_replica.cc:330] T 05b2485fa01a4706a85ebc150306637f P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:42.085397 1292 raft_consensus.cc:2229] T 05b2485fa01a4706a85ebc150306637f P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:42.085846 1292 raft_consensus.cc:2258] T 05b2485fa01a4706a85ebc150306637f P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:42.090631 1292 ts_tablet_manager.cc:1875] T 05b2485fa01a4706a85ebc150306637f P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:42.092442 1287 ts_tablet_manager.cc:1888] T 0be6dad1a409426c9a3dc0866505b3ea P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:42.097245 27758 raft_consensus.cc:1261] T 448a24d99bbe4fe4a571dc49a04ee275 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:42.098035 1108 consensus_queue.cc:1035] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:42.098253 1287 log.cc:1196] T 0be6dad1a409426c9a3dc0866505b3ea P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/0be6dad1a409426c9a3dc0866505b3ea
I20240501 08:12:42.099133 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet 0be6dad1a409426c9a3dc0866505b3ea (table rebalance_test_table_17 [id=1121d4c7d5444060ac6df36744663e41]) successfully deleted
I20240501 08:12:42.099210 30536 raft_consensus.cc:1261] T 448a24d99bbe4fe4a571dc49a04ee275 P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:42.095060 1261 tablet_copy_client.cc:783] T 87804550560d45269e6d4d6fe8938f55 P aad81047009a4ffcbc450b60030583b5: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:42.100098 1261 tablet_copy_client.cc:647] T 87804550560d45269e6d4d6fe8938f55 P aad81047009a4ffcbc450b60030583b5: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:42.101872 1261 tablet_copy_client.cc:523] T 87804550560d45269e6d4d6fe8938f55 P aad81047009a4ffcbc450b60030583b5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:42.103964 1286 tablet_copy_client.cc:308] T bb7dae019c7e4842b0daa35ee4b40367 P 31811708c6774037b82df44024b229b6: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.129:43471
I20240501 08:12:42.107036 27781 tablet_copy_service.cc:140] P 6331f77dae9d452184e8cd0cbf5f1b43: Received BeginTabletCopySession request for tablet bb7dae019c7e4842b0daa35ee4b40367 from peer 31811708c6774037b82df44024b229b6 ({username='slave'} at 127.21.84.133:35347)
I20240501 08:12:42.107226 27781 tablet_copy_service.cc:161] P 6331f77dae9d452184e8cd0cbf5f1b43: Beginning new tablet copy session on tablet bb7dae019c7e4842b0daa35ee4b40367 from peer 31811708c6774037b82df44024b229b6 at {username='slave'} at 127.21.84.133:35347: session id = 31811708c6774037b82df44024b229b6-bb7dae019c7e4842b0daa35ee4b40367
I20240501 08:12:42.109043 1292 ts_tablet_manager.cc:1888] T 05b2485fa01a4706a85ebc150306637f P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:42.109992 1292 log.cc:1196] T 05b2485fa01a4706a85ebc150306637f P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/05b2485fa01a4706a85ebc150306637f
I20240501 08:12:42.110491 27781 tablet_copy_source_session.cc:205] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:42.110911 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet 05b2485fa01a4706a85ebc150306637f (table rebalance_test_table_5 [id=b19758166c02438c9fc4cfa835acc028]) successfully deleted
I20240501 08:12:42.111300 362 consensus_queue.cc:1035] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20240501 08:12:42.123585 1261 tablet_bootstrap.cc:492] T 87804550560d45269e6d4d6fe8938f55 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:42.124825 1286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb7dae019c7e4842b0daa35ee4b40367. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:42.132216 1286 tablet_copy_client.cc:783] T bb7dae019c7e4842b0daa35ee4b40367 P 31811708c6774037b82df44024b229b6: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:42.133371 1286 tablet_copy_client.cc:647] T bb7dae019c7e4842b0daa35ee4b40367 P 31811708c6774037b82df44024b229b6: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:42.135413 1286 tablet_copy_client.cc:523] T bb7dae019c7e4842b0daa35ee4b40367 P 31811708c6774037b82df44024b229b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:42.140404 1286 tablet_bootstrap.cc:492] T bb7dae019c7e4842b0daa35ee4b40367 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:42.147293 30934 raft_consensus.cc:1261] T 448a24d99bbe4fe4a571dc49a04ee275 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:42.147960 362 consensus_queue.cc:1035] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:42.156432 1261 tablet_bootstrap.cc:492] T 87804550560d45269e6d4d6fe8938f55 P aad81047009a4ffcbc450b60030583b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:42.156821 1261 tablet_bootstrap.cc:492] T 87804550560d45269e6d4d6fe8938f55 P aad81047009a4ffcbc450b60030583b5: Bootstrap complete.
I20240501 08:12:42.157083 1261 ts_tablet_manager.cc:1367] T 87804550560d45269e6d4d6fe8938f55 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.004s
I20240501 08:12:42.158811 1261 raft_consensus.cc:348] T 87804550560d45269e6d4d6fe8938f55 P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:12:42.160198 1261 raft_consensus.cc:729] T 87804550560d45269e6d4d6fe8938f55 P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: LEARNER
W20240501 08:12:42.161310 1299 log.cc:924] Time spent T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20240501 08:12:42.161680 1261 consensus_queue.cc:260] T 87804550560d45269e6d4d6fe8938f55 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:12:42.162554 1233 raft_consensus.cc:2932] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
W20240501 08:12:42.167814 30473 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.130:36503, user_credentials={real_user=slave}} blocked reactor thread for 100857us
I20240501 08:12:42.175339 1261 ts_tablet_manager.cc:1398] T 87804550560d45269e6d4d6fe8938f55 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.018s user 0.002s sys 0.002s
I20240501 08:12:42.176136 30957 tablet_copy_service.cc:342] P a29cd590999b4f32849ff5689cd53c11: Request end of tablet copy session aad81047009a4ffcbc450b60030583b5-87804550560d45269e6d4d6fe8938f55 received from {username='slave'} at 127.21.84.131:60955
I20240501 08:12:42.176321 30957 tablet_copy_service.cc:433] P a29cd590999b4f32849ff5689cd53c11: ending tablet copy session aad81047009a4ffcbc450b60030583b5-87804550560d45269e6d4d6fe8938f55 on tablet 87804550560d45269e6d4d6fe8938f55 with peer aad81047009a4ffcbc450b60030583b5
I20240501 08:12:42.178522 1251 ts_tablet_manager.cc:902] T 95cb5060152e4237988a4c5723a62601 P 6331f77dae9d452184e8cd0cbf5f1b43: Initiating tablet copy from peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:42.186595 28022 raft_consensus.cc:1203] T 87804550560d45269e6d4d6fe8938f55 P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:42.187300 1176 raft_consensus.cc:1050] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11: attempting to promote NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 to VOTER
I20240501 08:12:42.188174 1176 consensus_queue.cc:237] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:12:42.203419 27760 raft_consensus.cc:1261] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:42.204140 456 consensus_queue.cc:1035] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:42.205685 30536 consensus_queue.cc:237] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:12:42.205991 30934 raft_consensus.cc:2932] T 448a24d99bbe4fe4a571dc49a04ee275 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:42.214957 1251 tablet_copy_client.cc:308] T 95cb5060152e4237988a4c5723a62601 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.131:36305
I20240501 08:12:42.215680 28042 tablet_copy_service.cc:140] P aad81047009a4ffcbc450b60030583b5: Received BeginTabletCopySession request for tablet 95cb5060152e4237988a4c5723a62601 from peer 6331f77dae9d452184e8cd0cbf5f1b43 ({username='slave'} at 127.21.84.129:39999)
I20240501 08:12:42.215819 28042 tablet_copy_service.cc:161] P aad81047009a4ffcbc450b60030583b5: Beginning new tablet copy session on tablet 95cb5060152e4237988a4c5723a62601 from peer 6331f77dae9d452184e8cd0cbf5f1b43 at {username='slave'} at 127.21.84.129:39999: session id = 6331f77dae9d452184e8cd0cbf5f1b43-95cb5060152e4237988a4c5723a62601
I20240501 08:12:42.216030 30534 raft_consensus.cc:2932] T 448a24d99bbe4fe4a571dc49a04ee275 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:42.218714 28042 tablet_copy_source_session.cc:205] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:42.224375 1251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95cb5060152e4237988a4c5723a62601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:42.227375 27616 catalog_manager.cc:5484] T 448a24d99bbe4fe4a571dc49a04ee275 P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index 2 to 3, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:42.229851 27758 raft_consensus.cc:2932] T 448a24d99bbe4fe4a571dc49a04ee275 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:42.232137 30800 raft_consensus.cc:1261] T 893c1555ca0849da8f48a4f253b42153 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:42.233065 523 consensus_queue.cc:1035] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:42.233974 28149 raft_consensus.cc:1261] T 893c1555ca0849da8f48a4f253b42153 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:42.235044 523 consensus_queue.cc:1035] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:12:42.242908 30472 consensus_peers.cc:485] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 -> Peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Couldn't send request to peer aad81047009a4ffcbc450b60030583b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 893c1555ca0849da8f48a4f253b42153. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:42.251938 1286 tablet_bootstrap.cc:492] T bb7dae019c7e4842b0daa35ee4b40367 P 31811708c6774037b82df44024b229b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:42.244860 1251 tablet_copy_client.cc:783] T 95cb5060152e4237988a4c5723a62601 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:42.252677 1251 tablet_copy_client.cc:647] T 95cb5060152e4237988a4c5723a62601 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:42.254997 1286 tablet_bootstrap.cc:492] T bb7dae019c7e4842b0daa35ee4b40367 P 31811708c6774037b82df44024b229b6: Bootstrap complete.
I20240501 08:12:42.255170 1251 tablet_copy_client.cc:523] T 95cb5060152e4237988a4c5723a62601 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:42.255319 1286 ts_tablet_manager.cc:1367] T bb7dae019c7e4842b0daa35ee4b40367 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.115s user 0.049s sys 0.010s
I20240501 08:12:42.256177 1286 raft_consensus.cc:348] T bb7dae019c7e4842b0daa35ee4b40367 P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
I20240501 08:12:42.256500 1286 raft_consensus.cc:729] T bb7dae019c7e4842b0daa35ee4b40367 P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: LEARNER
I20240501 08:12:42.256705 1286 consensus_queue.cc:260] T bb7dae019c7e4842b0daa35ee4b40367 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
I20240501 08:12:42.257428 1286 ts_tablet_manager.cc:1398] T bb7dae019c7e4842b0daa35ee4b40367 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 08:12:42.258807 1251 tablet_bootstrap.cc:492] T 95cb5060152e4237988a4c5723a62601 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:42.259702 27781 tablet_copy_service.cc:342] P 6331f77dae9d452184e8cd0cbf5f1b43: Request end of tablet copy session 31811708c6774037b82df44024b229b6-bb7dae019c7e4842b0daa35ee4b40367 received from {username='slave'} at 127.21.84.133:35347
I20240501 08:12:42.259806 27781 tablet_copy_service.cc:433] P 6331f77dae9d452184e8cd0cbf5f1b43: ending tablet copy session 31811708c6774037b82df44024b229b6-bb7dae019c7e4842b0daa35ee4b40367 on tablet bb7dae019c7e4842b0daa35ee4b40367 with peer 31811708c6774037b82df44024b229b6
I20240501 08:12:42.260457 523 raft_consensus.cc:2932] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:42.268906 30672 consensus_queue.cc:237] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } }
I20240501 08:12:42.270954 28149 raft_consensus.cc:2932] T 893c1555ca0849da8f48a4f253b42153 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:42.272238 30536 raft_consensus.cc:1261] T 448a24d99bbe4fe4a571dc49a04ee275 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:42.273983 1108 consensus_queue.cc:1035] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:42.278211 27612 catalog_manager.cc:5484] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:42.282604 27758 raft_consensus.cc:1203] T b5d7223054aa4308917fd402d65ded5d P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:42.290561 1108 raft_consensus.cc:2932] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:42.295920 30536 raft_consensus.cc:2932] T 448a24d99bbe4fe4a571dc49a04ee275 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:42.305975 27604 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 448a24d99bbe4fe4a571dc49a04ee275 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:42.309810 27611 catalog_manager.cc:5484] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 3 to 4, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New cstate: current_term: 1 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 08:12:42.310168 30737 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.132:41323, user_credentials={real_user=slave}} blocked reactor thread for 93964.2us
I20240501 08:12:42.314641 27741 tablet_service.cc:1514] Processing DeleteTablet for tablet 448a24d99bbe4fe4a571dc49a04ee275 with delete_type TABLET_DATA_TOMBSTONED (TS 6331f77dae9d452184e8cd0cbf5f1b43 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:42.324209 1251 tablet_bootstrap.cc:492] T 95cb5060152e4237988a4c5723a62601 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:42.324368 28151 raft_consensus.cc:1261] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:42.324656 1251 tablet_bootstrap.cc:492] T 95cb5060152e4237988a4c5723a62601 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap complete.
I20240501 08:12:42.324919 1251 ts_tablet_manager.cc:1367] T 95cb5060152e4237988a4c5723a62601 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.066s user 0.004s sys 0.007s
I20240501 08:12:42.326056 30672 raft_consensus.cc:1261] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:42.326017 1251 raft_consensus.cc:348] T 95cb5060152e4237988a4c5723a62601 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:12:42.326251 1251 raft_consensus.cc:729] T 95cb5060152e4237988a4c5723a62601 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: LEARNER
I20240501 08:12:42.326367 30934 raft_consensus.cc:1261] T 448a24d99bbe4fe4a571dc49a04ee275 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:42.326469 1251 consensus_queue.cc:260] T 95cb5060152e4237988a4c5723a62601 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:12:42.327088 1251 ts_tablet_manager.cc:1398] T 95cb5060152e4237988a4c5723a62601 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:12:42.329658 1233 consensus_queue.cc:1035] T 448a24d99bbe4fe4a571dc49a04ee275 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:42.330549 28042 tablet_copy_service.cc:342] P aad81047009a4ffcbc450b60030583b5: Request end of tablet copy session 6331f77dae9d452184e8cd0cbf5f1b43-95cb5060152e4237988a4c5723a62601 received from {username='slave'} at 127.21.84.129:39999
I20240501 08:12:42.330714 28042 tablet_copy_service.cc:433] P aad81047009a4ffcbc450b60030583b5: ending tablet copy session 6331f77dae9d452184e8cd0cbf5f1b43-95cb5060152e4237988a4c5723a62601 on tablet 95cb5060152e4237988a4c5723a62601 with peer 6331f77dae9d452184e8cd0cbf5f1b43
I20240501 08:12:42.330710 1325 consensus_queue.cc:1035] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:42.339864 1137 tablet_bootstrap.cc:492] T 4b3e3d39f82549079293d98357884bf0 P a29cd590999b4f32849ff5689cd53c11: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:42.340773 523 raft_consensus.cc:1050] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6: attempting to promote NON_VOTER a29cd590999b4f32849ff5689cd53c11 to VOTER
I20240501 08:12:42.341735 523 consensus_queue.cc:237] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:12:42.344269 1326 tablet_replica.cc:330] T 448a24d99bbe4fe4a571dc49a04ee275 P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:42.346432 27890 raft_consensus.cc:1261] T d020e1225d5d447f993f920569a05f3d P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:42.347158 939 consensus_queue.cc:1035] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:42.354327 28149 raft_consensus.cc:1261] T d020e1225d5d447f993f920569a05f3d P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:42.355144 1300 consensus_queue.cc:1035] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:42.356523 1326 raft_consensus.cc:2229] T 448a24d99bbe4fe4a571dc49a04ee275 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:42.344640 1325 consensus_queue.cc:1035] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:42.349793 30935 raft_consensus.cc:1261] T d020e1225d5d447f993f920569a05f3d P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:42.349269 30934 raft_consensus.cc:2932] T 448a24d99bbe4fe4a571dc49a04ee275 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:42.358417 1300 consensus_queue.cc:1035] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:42.363289 1137 tablet_bootstrap.cc:492] T 4b3e3d39f82549079293d98357884bf0 P a29cd590999b4f32849ff5689cd53c11: Bootstrap complete.
I20240501 08:12:42.364140 1326 raft_consensus.cc:2258] T 448a24d99bbe4fe4a571dc49a04ee275 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:42.365562 1137 ts_tablet_manager.cc:1367] T 4b3e3d39f82549079293d98357884bf0 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.363s user 0.070s sys 0.026s
I20240501 08:12:42.381249 30803 raft_consensus.cc:2932] T 893c1555ca0849da8f48a4f253b42153 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:42.383258 1137 raft_consensus.cc:348] T 4b3e3d39f82549079293d98357884bf0 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:12:42.385879 1326 ts_tablet_manager.cc:1875] T 448a24d99bbe4fe4a571dc49a04ee275 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:42.390254 1137 raft_consensus.cc:729] T 4b3e3d39f82549079293d98357884bf0 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: LEARNER
I20240501 08:12:42.390585 1137 consensus_queue.cc:260] T 4b3e3d39f82549079293d98357884bf0 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:12:42.391232 1137 ts_tablet_manager.cc:1398] T 4b3e3d39f82549079293d98357884bf0 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240501 08:12:42.404434 1087 raft_consensus.cc:1050] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43: attempting to promote NON_VOTER 64998323a1f4482d8704bf20594d13a5 to VOTER
I20240501 08:12:42.405364 1087 consensus_queue.cc:237] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:12:42.423941 1160 raft_consensus.cc:1050] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5: attempting to promote NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 to VOTER
I20240501 08:12:42.424772 1160 consensus_queue.cc:237] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:12:42.426378 28172 tablet_copy_service.cc:342] P 91d11eae5ef6432ca86eb8ed446b6ea6: Request end of tablet copy session a29cd590999b4f32849ff5689cd53c11-4b3e3d39f82549079293d98357884bf0 received from {username='slave'} at 127.21.84.136:39335
I20240501 08:12:42.426506 28172 tablet_copy_service.cc:433] P 91d11eae5ef6432ca86eb8ed446b6ea6: ending tablet copy session a29cd590999b4f32849ff5689cd53c11-4b3e3d39f82549079293d98357884bf0 on tablet 4b3e3d39f82549079293d98357884bf0 with peer a29cd590999b4f32849ff5689cd53c11
I20240501 08:12:42.432550 1296 raft_consensus.cc:2932] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:42.435005 28152 raft_consensus.cc:1261] T 2e1dd7a48a574ffcbea936af5b0fe192 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:42.436012 1319 consensus_queue.cc:1035] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:42.436990 30669 raft_consensus.cc:1261] T 2e1dd7a48a574ffcbea936af5b0fe192 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:42.437069 28022 raft_consensus.cc:1261] T 2e1dd7a48a574ffcbea936af5b0fe192 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:42.438616 1319 consensus_queue.cc:1035] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:42.439196 1319 consensus_queue.cc:1035] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:42.439289 27890 raft_consensus.cc:2932] T d020e1225d5d447f993f920569a05f3d P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:42.447863 27616 catalog_manager.cc:5484] T d020e1225d5d447f993f920569a05f3d P f1525cbc82a24d13b65f136695c903af reported cstate change: config changed from index 2 to 3, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:42.457654 28150 raft_consensus.cc:2932] T d020e1225d5d447f993f920569a05f3d P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:42.472409 27890 raft_consensus.cc:1261] T 32a54b70dcc447828ec293a9ff0afb80 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:42.473152 1087 consensus_queue.cc:1035] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:42.473454 30800 raft_consensus.cc:1261] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:42.474169 1089 consensus_queue.cc:1035] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:42.474546 28019 raft_consensus.cc:1261] T 32a54b70dcc447828ec293a9ff0afb80 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:42.475201 1089 consensus_queue.cc:1035] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:42.484282 1326 ts_tablet_manager.cc:1888] T 448a24d99bbe4fe4a571dc49a04ee275 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:42.489326 1326 log.cc:1196] T 448a24d99bbe4fe4a571dc49a04ee275 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/448a24d99bbe4fe4a571dc49a04ee275
I20240501 08:12:42.490201 28018 raft_consensus.cc:2932] T 2e1dd7a48a574ffcbea936af5b0fe192 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:42.495122 1154 raft_consensus.cc:2932] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:42.496866 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet 448a24d99bbe4fe4a571dc49a04ee275 (table rebalancer_extra_table_58 [id=7d3b03da535e4c14ad4122a4e27c19b5]) successfully deleted
I20240501 08:12:42.497924 30669 raft_consensus.cc:2932] T 2e1dd7a48a574ffcbea936af5b0fe192 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:42.505553 456 raft_consensus.cc:2932] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:42.510219 27758 raft_consensus.cc:2932] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:42.515712 30935 raft_consensus.cc:2932] T d020e1225d5d447f993f920569a05f3d P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:42.520998 27619 catalog_manager.cc:5484] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:42.521015 27618 catalog_manager.cc:5484] T 2e1dd7a48a574ffcbea936af5b0fe192 P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 2 to 3, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:42.530843 30534 raft_consensus.cc:1203] T bb7dae019c7e4842b0daa35ee4b40367 P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:42.535167 30534 consensus_queue.cc:237] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:12:42.537968 28152 raft_consensus.cc:1261] T d020e1225d5d447f993f920569a05f3d P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:42.538548 1296 consensus_queue.cc:1035] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:42.541035 1296 raft_consensus.cc:2932] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:42.549302 30672 raft_consensus.cc:2932] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:42.555209 28150 raft_consensus.cc:2932] T 2e1dd7a48a574ffcbea936af5b0fe192 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:42.557109 27607 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d020e1225d5d447f993f920569a05f3d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:42.568490 28150 raft_consensus.cc:2932] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:42.575842 1087 raft_consensus.cc:2932] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:42.576455 30803 raft_consensus.cc:2932] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:42.576797 30937 consensus_queue.cc:237] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:12:42.579353 27758 raft_consensus.cc:1261] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:42.580329 1176 consensus_queue.cc:1035] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:42.582238 27890 raft_consensus.cc:2932] T 32a54b70dcc447828ec293a9ff0afb80 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:42.584270 28022 raft_consensus.cc:2932] T 32a54b70dcc447828ec293a9ff0afb80 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:42.589184 27618 catalog_manager.cc:5484] T 32a54b70dcc447828ec293a9ff0afb80 P f1525cbc82a24d13b65f136695c903af reported cstate change: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:42.591149 27612 catalog_manager.cc:5484] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index 3 to 4, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:42.593353 30934 raft_consensus.cc:1261] T d020e1225d5d447f993f920569a05f3d P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:42.594144 1296 consensus_queue.cc:1035] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:42.596725 30803 consensus_queue.cc:237] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:12:42.601526 30935 raft_consensus.cc:2932] T d020e1225d5d447f993f920569a05f3d P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:42.602037 28022 raft_consensus.cc:1261] T 2e1dd7a48a574ffcbea936af5b0fe192 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:42.605773 1319 consensus_queue.cc:1035] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:42.606009 1324 raft_consensus.cc:2932] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:42.607359 27871 tablet_service.cc:1514] Processing DeleteTablet for tablet d020e1225d5d447f993f920569a05f3d with delete_type TABLET_DATA_TOMBSTONED (TS f1525cbc82a24d13b65f136695c903af not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:42.607703 1292 tablet_replica.cc:330] T d020e1225d5d447f993f920569a05f3d P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:42.607923 1292 raft_consensus.cc:2229] T d020e1225d5d447f993f920569a05f3d P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:42.608186 1292 raft_consensus.cc:2258] T d020e1225d5d447f993f920569a05f3d P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:42.610637 1292 ts_tablet_manager.cc:1875] T d020e1225d5d447f993f920569a05f3d P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:42.612463 28152 raft_consensus.cc:2932] T d020e1225d5d447f993f920569a05f3d P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:42.613373 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8bbd50f618d14aad8e45c2aa6fbabc1f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:42.614758 28151 raft_consensus.cc:1261] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:42.615809 456 consensus_queue.cc:1035] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:42.617733 28151 raft_consensus.cc:2932] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:42.632728 1154 raft_consensus.cc:2932] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:42.634326 27758 raft_consensus.cc:2932] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:42.637024 27605 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2e1dd7a48a574ffcbea936af5b0fe192 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:42.638353 27619 catalog_manager.cc:5484] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:42.639153 27760 raft_consensus.cc:1203] T 95cb5060152e4237988a4c5723a62601 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:42.644573 1346 ts_tablet_manager.cc:902] T 4f8c0a8e86f04f32b79743f819948544 P f1525cbc82a24d13b65f136695c903af: Initiating tablet copy from peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:42.652910 1292 ts_tablet_manager.cc:1888] T d020e1225d5d447f993f920569a05f3d P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:42.653185 1292 log.cc:1196] T d020e1225d5d447f993f920569a05f3d P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/d020e1225d5d447f993f920569a05f3d
I20240501 08:12:42.653281 27760 consensus_queue.cc:237] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:12:42.654251 28022 raft_consensus.cc:2932] T 2e1dd7a48a574ffcbea936af5b0fe192 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:42.655522 30935 raft_consensus.cc:1203] T 4b3e3d39f82549079293d98357884bf0 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:42.656078 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet d020e1225d5d447f993f920569a05f3d (table rebalancer_extra_table_56 [id=2e283cf00add44fc94e82872e4b0aab8]) successfully deleted
I20240501 08:12:42.657166 27890 raft_consensus.cc:1261] T 32a54b70dcc447828ec293a9ff0afb80 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:42.657963 1087 consensus_queue.cc:1035] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:42.664064 1087 raft_consensus.cc:2932] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:42.665215 30669 raft_consensus.cc:1261] T 2e1dd7a48a574ffcbea936af5b0fe192 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:42.670385 27891 raft_consensus.cc:2932] T 32a54b70dcc447828ec293a9ff0afb80 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:42.673605 1335 consensus_queue.cc:1035] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:42.675467 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 32a54b70dcc447828ec293a9ff0afb80 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:42.676229 30803 raft_consensus.cc:1261] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:42.677574 312 consensus_queue.cc:1035] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:42.683995 30803 raft_consensus.cc:2932] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:42.684979 27611 catalog_manager.cc:5484] T 32a54b70dcc447828ec293a9ff0afb80 P f1525cbc82a24d13b65f136695c903af reported cstate change: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:42.683562 30669 raft_consensus.cc:2932] T 2e1dd7a48a574ffcbea936af5b0fe192 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:42.691299 30650 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bbd50f618d14aad8e45c2aa6fbabc1f with delete_type TABLET_DATA_TOMBSTONED (TS 9a022e1a501f4d44a0468054d6c3de47 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57156
I20240501 08:12:42.703373 28000 tablet_service.cc:1514] Processing DeleteTablet for tablet 32a54b70dcc447828ec293a9ff0afb80 with delete_type TABLET_DATA_TOMBSTONED (TS aad81047009a4ffcbc450b60030583b5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:42.704422 1350 tablet_replica.cc:330] T 32a54b70dcc447828ec293a9ff0afb80 P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:42.704638 1350 raft_consensus.cc:2229] T 32a54b70dcc447828ec293a9ff0afb80 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:42.704836 1350 raft_consensus.cc:2258] T 32a54b70dcc447828ec293a9ff0afb80 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:42.716358 1350 ts_tablet_manager.cc:1875] T 32a54b70dcc447828ec293a9ff0afb80 P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:42.669531 27616 catalog_manager.cc:5484] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index 3 to 4, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:42.736439 1350 ts_tablet_manager.cc:1888] T 32a54b70dcc447828ec293a9ff0afb80 P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:42.737650 28131 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e1dd7a48a574ffcbea936af5b0fe192 with delete_type TABLET_DATA_TOMBSTONED (TS 91d11eae5ef6432ca86eb8ed446b6ea6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:42.738418 1350 log.cc:1196] T 32a54b70dcc447828ec293a9ff0afb80 P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/32a54b70dcc447828ec293a9ff0afb80
I20240501 08:12:42.744397 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet 32a54b70dcc447828ec293a9ff0afb80 (table rebalance_test_table_15 [id=d3eb6ca7d8cf4bfdb114edb23abb2ed2]) successfully deleted
I20240501 08:12:42.755738 1351 tablet_replica.cc:330] T 2e1dd7a48a574ffcbea936af5b0fe192 P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:42.756110 1351 raft_consensus.cc:2229] T 2e1dd7a48a574ffcbea936af5b0fe192 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:42.756422 1351 raft_consensus.cc:2258] T 2e1dd7a48a574ffcbea936af5b0fe192 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:42.762257 1349 tablet_replica.cc:330] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 9a022e1a501f4d44a0468054d6c3de47: stopping tablet replica
I20240501 08:12:42.762156 1351 ts_tablet_manager.cc:1875] T 2e1dd7a48a574ffcbea936af5b0fe192 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:42.762728 1349 raft_consensus.cc:2229] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:42.763851 1349 raft_consensus.cc:2258] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:42.768874 1351 ts_tablet_manager.cc:1888] T 2e1dd7a48a574ffcbea936af5b0fe192 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:42.772073 1349 ts_tablet_manager.cc:1875] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 9a022e1a501f4d44a0468054d6c3de47: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:42.780161 1349 ts_tablet_manager.cc:1888] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 9a022e1a501f4d44a0468054d6c3de47: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:42.780457 1349 log.cc:1196] T 8bbd50f618d14aad8e45c2aa6fbabc1f P 9a022e1a501f4d44a0468054d6c3de47: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/wals/8bbd50f618d14aad8e45c2aa6fbabc1f
I20240501 08:12:42.781615 27604 catalog_manager.cc:4830] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): tablet 8bbd50f618d14aad8e45c2aa6fbabc1f (table rebalancer_extra_table_24 [id=2fd11fb0500a494a804ac62257c15bf4]) successfully deleted
I20240501 08:12:42.782842 1351 log.cc:1196] T 2e1dd7a48a574ffcbea936af5b0fe192 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/2e1dd7a48a574ffcbea936af5b0fe192
I20240501 08:12:42.783752 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet 2e1dd7a48a574ffcbea936af5b0fe192 (table rebalancer_extra_table_52 [id=238033141b1b4b5b847f037773a4022f]) successfully deleted
I20240501 08:12:42.784700 1359 ts_tablet_manager.cc:902] T 893c1555ca0849da8f48a4f253b42153 P aad81047009a4ffcbc450b60030583b5: Initiating tablet copy from peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:42.801403 1346 tablet_copy_client.cc:308] T 4f8c0a8e86f04f32b79743f819948544 P f1525cbc82a24d13b65f136695c903af: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.133:38293
I20240501 08:12:42.802204 30539 tablet_copy_service.cc:140] P 31811708c6774037b82df44024b229b6: Received BeginTabletCopySession request for tablet 4f8c0a8e86f04f32b79743f819948544 from peer f1525cbc82a24d13b65f136695c903af ({username='slave'} at 127.21.84.130:58953)
I20240501 08:12:42.802371 30539 tablet_copy_service.cc:161] P 31811708c6774037b82df44024b229b6: Beginning new tablet copy session on tablet 4f8c0a8e86f04f32b79743f819948544 from peer f1525cbc82a24d13b65f136695c903af at {username='slave'} at 127.21.84.130:58953: session id = f1525cbc82a24d13b65f136695c903af-4f8c0a8e86f04f32b79743f819948544
I20240501 08:12:42.805333 30539 tablet_copy_source_session.cc:205] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:42.806897 1346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f8c0a8e86f04f32b79743f819948544. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:42.807988 456 raft_consensus.cc:1050] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11: attempting to promote NON_VOTER aad81047009a4ffcbc450b60030583b5 to VOTER
I20240501 08:12:42.808847 456 consensus_queue.cc:237] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } }
I20240501 08:12:42.816359 1359 tablet_copy_client.cc:308] T 893c1555ca0849da8f48a4f253b42153 P aad81047009a4ffcbc450b60030583b5: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.133:38293
I20240501 08:12:42.817181 30539 tablet_copy_service.cc:140] P 31811708c6774037b82df44024b229b6: Received BeginTabletCopySession request for tablet 893c1555ca0849da8f48a4f253b42153 from peer aad81047009a4ffcbc450b60030583b5 ({username='slave'} at 127.21.84.131:56895)
I20240501 08:12:42.817373 30539 tablet_copy_service.cc:161] P 31811708c6774037b82df44024b229b6: Beginning new tablet copy session on tablet 893c1555ca0849da8f48a4f253b42153 from peer aad81047009a4ffcbc450b60030583b5 at {username='slave'} at 127.21.84.131:56895: session id = aad81047009a4ffcbc450b60030583b5-893c1555ca0849da8f48a4f253b42153
I20240501 08:12:42.817687 1346 tablet_copy_client.cc:783] T 4f8c0a8e86f04f32b79743f819948544 P f1525cbc82a24d13b65f136695c903af: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:42.818213 1346 tablet_copy_client.cc:647] T 4f8c0a8e86f04f32b79743f819948544 P f1525cbc82a24d13b65f136695c903af: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:42.818279 28022 raft_consensus.cc:1261] T 87804550560d45269e6d4d6fe8938f55 P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:42.818820 1325 consensus_queue.cc:1035] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:42.820174 30539 tablet_copy_source_session.cc:205] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:42.820662 27891 raft_consensus.cc:1261] T 87804550560d45269e6d4d6fe8938f55 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:42.821622 1346 tablet_copy_client.cc:523] T 4f8c0a8e86f04f32b79743f819948544 P f1525cbc82a24d13b65f136695c903af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:42.822059 1325 consensus_queue.cc:1035] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:42.824898 1359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 893c1555ca0849da8f48a4f253b42153. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:42.825634 1346 tablet_bootstrap.cc:492] T 4f8c0a8e86f04f32b79743f819948544 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:42.828404 30803 raft_consensus.cc:1261] T 87804550560d45269e6d4d6fe8938f55 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:42.829181 1325 consensus_queue.cc:1035] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:42.841859 1325 raft_consensus.cc:2932] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:42.849701 28019 raft_consensus.cc:2932] T 87804550560d45269e6d4d6fe8938f55 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:42.850769 27891 raft_consensus.cc:2932] T 87804550560d45269e6d4d6fe8938f55 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:42.852641 30803 raft_consensus.cc:2932] T 87804550560d45269e6d4d6fe8938f55 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:42.853969 1359 tablet_copy_client.cc:783] T 893c1555ca0849da8f48a4f253b42153 P aad81047009a4ffcbc450b60030583b5: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:42.854247 1359 tablet_copy_client.cc:647] T 893c1555ca0849da8f48a4f253b42153 P aad81047009a4ffcbc450b60030583b5: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:42.857545 27612 catalog_manager.cc:5484] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:42.862110 456 raft_consensus.cc:1050] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11: attempting to promote NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 to VOTER
I20240501 08:12:42.862995 456 consensus_queue.cc:237] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:12:42.863678 1359 tablet_copy_client.cc:523] T 893c1555ca0849da8f48a4f253b42153 P aad81047009a4ffcbc450b60030583b5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:42.864729 1346 tablet_bootstrap.cc:492] T 4f8c0a8e86f04f32b79743f819948544 P f1525cbc82a24d13b65f136695c903af: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:42.864923 30935 consensus_queue.cc:237] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } }
I20240501 08:12:42.865203 1346 tablet_bootstrap.cc:492] T 4f8c0a8e86f04f32b79743f819948544 P f1525cbc82a24d13b65f136695c903af: Bootstrap complete.
I20240501 08:12:42.865449 1346 ts_tablet_manager.cc:1367] T 4f8c0a8e86f04f32b79743f819948544 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.040s user 0.014s sys 0.011s
I20240501 08:12:42.867041 28019 raft_consensus.cc:1261] T 87804550560d45269e6d4d6fe8938f55 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:42.867686 1324 consensus_queue.cc:1035] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:42.868161 1359 tablet_bootstrap.cc:492] T 893c1555ca0849da8f48a4f253b42153 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:42.866261 1346 raft_consensus.cc:348] T 4f8c0a8e86f04f32b79743f819948544 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:42.869172 1346 raft_consensus.cc:729] T 4f8c0a8e86f04f32b79743f819948544 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: LEARNER
I20240501 08:12:42.869392 1346 consensus_queue.cc:260] T 4f8c0a8e86f04f32b79743f819948544 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:42.871330 27890 raft_consensus.cc:1261] T 87804550560d45269e6d4d6fe8938f55 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:42.872252 1324 consensus_queue.cc:1035] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:42.875211 1346 ts_tablet_manager.cc:1398] T 4f8c0a8e86f04f32b79743f819948544 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240501 08:12:42.874895 1176 raft_consensus.cc:2932] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:42.877501 30559 tablet_copy_service.cc:342] P 31811708c6774037b82df44024b229b6: Request end of tablet copy session f1525cbc82a24d13b65f136695c903af-4f8c0a8e86f04f32b79743f819948544 received from {username='slave'} at 127.21.84.130:58953
I20240501 08:12:42.877637 30559 tablet_copy_service.cc:433] P 31811708c6774037b82df44024b229b6: ending tablet copy session f1525cbc82a24d13b65f136695c903af-4f8c0a8e86f04f32b79743f819948544 on tablet 4f8c0a8e86f04f32b79743f819948544 with peer f1525cbc82a24d13b65f136695c903af
I20240501 08:12:42.881956 1359 tablet_bootstrap.cc:492] T 893c1555ca0849da8f48a4f253b42153 P aad81047009a4ffcbc450b60030583b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:42.882014 30669 raft_consensus.cc:1261] T b5d7223054aa4308917fd402d65ded5d P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:42.882524 1359 tablet_bootstrap.cc:492] T 893c1555ca0849da8f48a4f253b42153 P aad81047009a4ffcbc450b60030583b5: Bootstrap complete.
I20240501 08:12:42.882786 1359 ts_tablet_manager.cc:1367] T 893c1555ca0849da8f48a4f253b42153 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20240501 08:12:42.883075 30536 raft_consensus.cc:1261] T b5d7223054aa4308917fd402d65ded5d P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:42.883517 1359 raft_consensus.cc:348] T 893c1555ca0849da8f48a4f253b42153 P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:12:42.883718 1359 raft_consensus.cc:729] T 893c1555ca0849da8f48a4f253b42153 P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: LEARNER
I20240501 08:12:42.883921 1359 consensus_queue.cc:260] T 893c1555ca0849da8f48a4f253b42153 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:12:42.884049 1325 consensus_queue.cc:1035] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:42.884912 27891 raft_consensus.cc:2932] T 87804550560d45269e6d4d6fe8938f55 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:42.886314 27758 raft_consensus.cc:1261] T b5d7223054aa4308917fd402d65ded5d P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:42.886380 1359 ts_tablet_manager.cc:1398] T 893c1555ca0849da8f48a4f253b42153 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:42.887059 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 87804550560d45269e6d4d6fe8938f55 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:42.889576 30559 tablet_copy_service.cc:342] P 31811708c6774037b82df44024b229b6: Request end of tablet copy session aad81047009a4ffcbc450b60030583b5-893c1555ca0849da8f48a4f253b42153 received from {username='slave'} at 127.21.84.131:56895
I20240501 08:12:42.889707 30559 tablet_copy_service.cc:433] P 31811708c6774037b82df44024b229b6: ending tablet copy session aad81047009a4ffcbc450b60030583b5-893c1555ca0849da8f48a4f253b42153 on tablet 893c1555ca0849da8f48a4f253b42153 with peer aad81047009a4ffcbc450b60030583b5
I20240501 08:12:42.887801 1324 consensus_queue.cc:1035] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20240501 08:12:42.892024 28019 raft_consensus.cc:2932] T 87804550560d45269e6d4d6fe8938f55 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:42.893380 27618 catalog_manager.cc:5484] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index 3 to 4, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:42.899657 1325 consensus_queue.cc:1035] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:42.903321 30784 tablet_service.cc:1514] Processing DeleteTablet for tablet 87804550560d45269e6d4d6fe8938f55 with delete_type TABLET_DATA_TOMBSTONED (TS 64998323a1f4482d8704bf20594d13a5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38952
I20240501 08:12:42.940515 1383 tablet_replica.cc:330] T 87804550560d45269e6d4d6fe8938f55 P 64998323a1f4482d8704bf20594d13a5: stopping tablet replica
I20240501 08:12:42.943224 1383 raft_consensus.cc:2229] T 87804550560d45269e6d4d6fe8938f55 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:42.944952 1383 raft_consensus.cc:2258] T 87804550560d45269e6d4d6fe8938f55 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:42.944940 1325 raft_consensus.cc:2932] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:42.945709 30669 raft_consensus.cc:2932] T b5d7223054aa4308917fd402d65ded5d P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:42.946638 1383 ts_tablet_manager.cc:1875] T 87804550560d45269e6d4d6fe8938f55 P 64998323a1f4482d8704bf20594d13a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:42.952831 27612 catalog_manager.cc:5484] T b5d7223054aa4308917fd402d65ded5d P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:42.954133 27758 raft_consensus.cc:2932] T b5d7223054aa4308917fd402d65ded5d P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:42.957794 30536 raft_consensus.cc:2932] T b5d7223054aa4308917fd402d65ded5d P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:42.966436 30935 consensus_queue.cc:237] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:12:42.969779 30670 raft_consensus.cc:1261] T b5d7223054aa4308917fd402d65ded5d P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:42.972965 1378 consensus_queue.cc:1035] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 08:12:42.974079 27760 raft_consensus.cc:1261] T b5d7223054aa4308917fd402d65ded5d P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:42.974941 1378 consensus_queue.cc:1035] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:42.976965 1379 raft_consensus.cc:2932] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:42.979010 27760 raft_consensus.cc:2932] T b5d7223054aa4308917fd402d65ded5d P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:42.981312 1383 ts_tablet_manager.cc:1888] T 87804550560d45269e6d4d6fe8938f55 P 64998323a1f4482d8704bf20594d13a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:42.981453 1383 log.cc:1196] T 87804550560d45269e6d4d6fe8938f55 P 64998323a1f4482d8704bf20594d13a5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal/wals/87804550560d45269e6d4d6fe8938f55
I20240501 08:12:42.982380 27605 catalog_manager.cc:4830] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): tablet 87804550560d45269e6d4d6fe8938f55 (table rebalancer_extra_table_40 [id=b218cf6d6a9e4117abbfea2879ddcd54]) successfully deleted
I20240501 08:12:42.982661 30670 raft_consensus.cc:2932] T b5d7223054aa4308917fd402d65ded5d P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:42.987579 27618 catalog_manager.cc:5484] T b5d7223054aa4308917fd402d65ded5d P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:42.988768 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b5d7223054aa4308917fd402d65ded5d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:42.992939 30514 tablet_service.cc:1514] Processing DeleteTablet for tablet b5d7223054aa4308917fd402d65ded5d with delete_type TABLET_DATA_TOMBSTONED (TS 31811708c6774037b82df44024b229b6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45078
I20240501 08:12:43.010541 1391 tablet_replica.cc:330] T b5d7223054aa4308917fd402d65ded5d P 31811708c6774037b82df44024b229b6: stopping tablet replica
I20240501 08:12:43.011876 1391 raft_consensus.cc:2229] T b5d7223054aa4308917fd402d65ded5d P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:43.012236 1391 raft_consensus.cc:2258] T b5d7223054aa4308917fd402d65ded5d P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:43.013638 1391 ts_tablet_manager.cc:1875] T b5d7223054aa4308917fd402d65ded5d P 31811708c6774037b82df44024b229b6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:43.026371 1391 ts_tablet_manager.cc:1888] T b5d7223054aa4308917fd402d65ded5d P 31811708c6774037b82df44024b229b6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:43.026662 1391 log.cc:1196] T b5d7223054aa4308917fd402d65ded5d P 31811708c6774037b82df44024b229b6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/wals/b5d7223054aa4308917fd402d65ded5d
I20240501 08:12:43.027525 27607 catalog_manager.cc:4830] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): tablet b5d7223054aa4308917fd402d65ded5d (table rebalancer_extra_table_12 [id=7c8060b6145d492cadb48df007ce38c9]) successfully deleted
I20240501 08:12:43.088351 1244 raft_consensus.cc:1050] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5: attempting to promote NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 to VOTER
I20240501 08:12:43.089175 1244 consensus_queue.cc:237] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:12:43.103034 27758 raft_consensus.cc:1261] T 95cb5060152e4237988a4c5723a62601 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:43.103886 877 consensus_queue.cc:1035] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:43.113550 30670 raft_consensus.cc:1261] T 95cb5060152e4237988a4c5723a62601 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:43.114212 877 consensus_queue.cc:1035] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:43.127996 30934 raft_consensus.cc:1261] T 95cb5060152e4237988a4c5723a62601 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:43.128614 877 consensus_queue.cc:1035] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:43.134070 1246 raft_consensus.cc:2932] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:43.140270 30669 raft_consensus.cc:2932] T 95cb5060152e4237988a4c5723a62601 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:43.145730 27760 raft_consensus.cc:2932] T 95cb5060152e4237988a4c5723a62601 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:43.149904 27619 catalog_manager.cc:5484] T 95cb5060152e4237988a4c5723a62601 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:43.155829 1089 raft_consensus.cc:1050] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43: attempting to promote NON_VOTER 31811708c6774037b82df44024b229b6 to VOTER
I20240501 08:12:43.159682 1089 consensus_queue.cc:237] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } }
I20240501 08:12:43.177686 28020 raft_consensus.cc:1261] T bb7dae019c7e4842b0daa35ee4b40367 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:43.181723 28149 raft_consensus.cc:1261] T bb7dae019c7e4842b0daa35ee4b40367 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:43.183224 30934 raft_consensus.cc:2932] T 95cb5060152e4237988a4c5723a62601 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:43.185817 1427 consensus_queue.cc:1035] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:43.197458 1429 consensus_queue.cc:1035] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:43.199554 28019 consensus_queue.cc:237] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:12:43.205000 27758 raft_consensus.cc:1261] T 95cb5060152e4237988a4c5723a62601 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:43.205005 30934 raft_consensus.cc:1261] T 95cb5060152e4237988a4c5723a62601 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:43.209569 1244 consensus_queue.cc:1035] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:43.214521 27891 raft_consensus.cc:1203] T 4f8c0a8e86f04f32b79743f819948544 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:43.215801 1244 consensus_queue.cc:1035] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:43.217499 877 raft_consensus.cc:2932] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:43.222520 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 95cb5060152e4237988a4c5723a62601 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:43.223543 1427 raft_consensus.cc:2932] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:43.226130 30536 raft_consensus.cc:1261] T bb7dae019c7e4842b0daa35ee4b40367 P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:43.228143 1429 consensus_queue.cc:1035] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:43.224133 28019 raft_consensus.cc:2932] T bb7dae019c7e4842b0daa35ee4b40367 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:43.230719 27619 catalog_manager.cc:5484] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:43.236200 30650 tablet_service.cc:1514] Processing DeleteTablet for tablet 95cb5060152e4237988a4c5723a62601 with delete_type TABLET_DATA_TOMBSTONED (TS 9a022e1a501f4d44a0468054d6c3de47 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57156
I20240501 08:12:43.236644 1349 tablet_replica.cc:330] T 95cb5060152e4237988a4c5723a62601 P 9a022e1a501f4d44a0468054d6c3de47: stopping tablet replica
I20240501 08:12:43.236934 1349 raft_consensus.cc:2229] T 95cb5060152e4237988a4c5723a62601 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:43.237296 1349 raft_consensus.cc:2258] T 95cb5060152e4237988a4c5723a62601 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:43.238723 1349 ts_tablet_manager.cc:1875] T 95cb5060152e4237988a4c5723a62601 P 9a022e1a501f4d44a0468054d6c3de47: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:43.248883 1349 ts_tablet_manager.cc:1888] T 95cb5060152e4237988a4c5723a62601 P 9a022e1a501f4d44a0468054d6c3de47: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:43.249210 1349 log.cc:1196] T 95cb5060152e4237988a4c5723a62601 P 9a022e1a501f4d44a0468054d6c3de47: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/wals/95cb5060152e4237988a4c5723a62601
I20240501 08:12:43.250782 30934 raft_consensus.cc:2932] T 95cb5060152e4237988a4c5723a62601 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:43.260653 27758 raft_consensus.cc:2932] T 95cb5060152e4237988a4c5723a62601 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:43.270071 840 raft_consensus.cc:1050] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6: attempting to promote NON_VOTER a29cd590999b4f32849ff5689cd53c11 to VOTER
I20240501 08:12:43.271075 840 consensus_queue.cc:237] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:12:43.291206 28020 raft_consensus.cc:1203] T 893c1555ca0849da8f48a4f253b42153 P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:43.299111 27604 catalog_manager.cc:4830] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): tablet 95cb5060152e4237988a4c5723a62601 (table rebalancer_extra_table_34 [id=1d72487b1d1a4310bbfd3ad3afddbee9]) successfully deleted
I20240501 08:12:43.300475 30536 raft_consensus.cc:2932] T bb7dae019c7e4842b0daa35ee4b40367 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:43.312752 28151 raft_consensus.cc:2932] T bb7dae019c7e4842b0daa35ee4b40367 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:43.316114 27618 catalog_manager.cc:5484] T bb7dae019c7e4842b0daa35ee4b40367 P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index 4 to 5, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:43.339668 30934 raft_consensus.cc:1261] T 4b3e3d39f82549079293d98357884bf0 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:43.340505 27758 consensus_queue.cc:237] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } }
I20240501 08:12:43.341840 28019 raft_consensus.cc:1261] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:43.342046 840 consensus_queue.cc:1035] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:43.342710 1199 consensus_queue.cc:1035] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:43.343328 30536 raft_consensus.cc:1261] T bb7dae019c7e4842b0daa35ee4b40367 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:43.343439 28019 raft_consensus.cc:1261] T bb7dae019c7e4842b0daa35ee4b40367 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:43.343863 312 consensus_queue.cc:1035] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:43.344183 1432 consensus_queue.cc:1035] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:43.347231 27891 raft_consensus.cc:1261] T 4b3e3d39f82549079293d98357884bf0 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:43.346951 1432 raft_consensus.cc:2932] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:43.349588 840 consensus_queue.cc:1035] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:43.357831 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bb7dae019c7e4842b0daa35ee4b40367 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:43.359422 1435 raft_consensus.cc:2932] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:43.360359 28022 raft_consensus.cc:2932] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:43.368124 27891 raft_consensus.cc:2932] T 4b3e3d39f82549079293d98357884bf0 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:43.371083 27618 catalog_manager.cc:5484] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 08:12:43.374984 27606 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet bb7dae019c7e4842b0daa35ee4b40367 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 5 but the committed config has opid_index of 6
I20240501 08:12:43.375890 28020 raft_consensus.cc:2932] T bb7dae019c7e4842b0daa35ee4b40367 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:43.377892 30536 raft_consensus.cc:2932] T bb7dae019c7e4842b0daa35ee4b40367 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:43.379554 27619 catalog_manager.cc:5484] T bb7dae019c7e4842b0daa35ee4b40367 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index 5 to 6, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:43.384630 28131 tablet_service.cc:1514] Processing DeleteTablet for tablet bb7dae019c7e4842b0daa35ee4b40367 with delete_type TABLET_DATA_TOMBSTONED (TS 91d11eae5ef6432ca86eb8ed446b6ea6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:43.419970 1442 tablet_replica.cc:330] T bb7dae019c7e4842b0daa35ee4b40367 P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:43.420707 1442 raft_consensus.cc:2229] T bb7dae019c7e4842b0daa35ee4b40367 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:43.421159 1442 raft_consensus.cc:2258] T bb7dae019c7e4842b0daa35ee4b40367 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:43.422756 30934 raft_consensus.cc:2932] T 4b3e3d39f82549079293d98357884bf0 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:43.432359 28151 consensus_queue.cc:237] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:12:43.439610 28020 raft_consensus.cc:1261] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:43.439652 30935 raft_consensus.cc:1261] T 4b3e3d39f82549079293d98357884bf0 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:43.440264 1199 consensus_queue.cc:1035] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:43.440798 1199 consensus_queue.cc:1035] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:43.444324 1435 raft_consensus.cc:2932] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:43.448838 27604 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4b3e3d39f82549079293d98357884bf0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:43.450845 30935 raft_consensus.cc:2932] T 4b3e3d39f82549079293d98357884bf0 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:43.454617 28019 raft_consensus.cc:2932] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:43.462965 27612 catalog_manager.cc:5484] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 5 to 6, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:43.470757 27871 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b3e3d39f82549079293d98357884bf0 with delete_type TABLET_DATA_TOMBSTONED (TS f1525cbc82a24d13b65f136695c903af not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:43.482844 1442 ts_tablet_manager.cc:1875] T bb7dae019c7e4842b0daa35ee4b40367 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:43.495204 1444 tablet_replica.cc:330] T 4b3e3d39f82549079293d98357884bf0 P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:43.496856 1444 raft_consensus.cc:2229] T 4b3e3d39f82549079293d98357884bf0 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:43.497226 1444 raft_consensus.cc:2258] T 4b3e3d39f82549079293d98357884bf0 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:43.498337 1442 ts_tablet_manager.cc:1888] T bb7dae019c7e4842b0daa35ee4b40367 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:43.498463 1442 log.cc:1196] T bb7dae019c7e4842b0daa35ee4b40367 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/bb7dae019c7e4842b0daa35ee4b40367
I20240501 08:12:43.499751 1444 ts_tablet_manager.cc:1875] T 4b3e3d39f82549079293d98357884bf0 P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:43.499890 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet bb7dae019c7e4842b0daa35ee4b40367 (table rebalance_test_table_13 [id=86d1b715252040a3bc0f714b3db3f4ff]) successfully deleted
I20240501 08:12:43.507242 1444 ts_tablet_manager.cc:1888] T 4b3e3d39f82549079293d98357884bf0 P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:43.507462 1444 log.cc:1196] T 4b3e3d39f82549079293d98357884bf0 P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/4b3e3d39f82549079293d98357884bf0
I20240501 08:12:43.508450 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet 4b3e3d39f82549079293d98357884bf0 (table rebalance_test_table_3 [id=56fa27c3d3de4793aadbcaf54cdb0ac5]) successfully deleted
I20240501 08:12:43.572753 1296 raft_consensus.cc:1050] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6: attempting to promote NON_VOTER f1525cbc82a24d13b65f136695c903af to VOTER
I20240501 08:12:43.574628 1296 consensus_queue.cc:237] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:43.577435 30935 raft_consensus.cc:1261] T 4f8c0a8e86f04f32b79743f819948544 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:43.578598 27890 raft_consensus.cc:1261] T 4f8c0a8e86f04f32b79743f819948544 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:43.578608 939 consensus_queue.cc:1035] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:43.579052 28019 raft_consensus.cc:1261] T 4f8c0a8e86f04f32b79743f819948544 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:43.580647 1300 consensus_queue.cc:1035] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:43.581336 1300 consensus_queue.cc:1035] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:43.594493 939 raft_consensus.cc:2932] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:43.597534 30935 raft_consensus.cc:2932] T 4f8c0a8e86f04f32b79743f819948544 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:43.610672 27890 raft_consensus.cc:2932] T 4f8c0a8e86f04f32b79743f819948544 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:43.612962 27612 catalog_manager.cc:5484] T 4f8c0a8e86f04f32b79743f819948544 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:43.625615 28019 raft_consensus.cc:2932] T 4f8c0a8e86f04f32b79743f819948544 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:43.655465 30537 consensus_queue.cc:237] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:43.657883 28019 raft_consensus.cc:1261] T 4f8c0a8e86f04f32b79743f819948544 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:43.659399 1300 consensus_queue.cc:1035] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:43.662693 27891 raft_consensus.cc:1261] T 4f8c0a8e86f04f32b79743f819948544 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:43.663245 1300 consensus_queue.cc:1035] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:43.665577 939 raft_consensus.cc:2932] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:43.666289 27890 raft_consensus.cc:2932] T 4f8c0a8e86f04f32b79743f819948544 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:43.671368 27607 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4f8c0a8e86f04f32b79743f819948544 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:43.671277 28019 raft_consensus.cc:2932] T 4f8c0a8e86f04f32b79743f819948544 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:43.675700 1296 raft_consensus.cc:1050] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6: attempting to promote NON_VOTER aad81047009a4ffcbc450b60030583b5 to VOTER
I20240501 08:12:43.676009 27618 catalog_manager.cc:5484] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:43.676928 1296 consensus_queue.cc:237] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } }
I20240501 08:12:43.682320 28020 raft_consensus.cc:1261] T 893c1555ca0849da8f48a4f253b42153 P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:43.682484 30800 raft_consensus.cc:1261] T 893c1555ca0849da8f48a4f253b42153 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:43.682971 28151 raft_consensus.cc:1261] T 893c1555ca0849da8f48a4f253b42153 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:43.683614 1300 consensus_queue.cc:1035] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:43.683969 939 consensus_queue.cc:1035] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:43.689900 30916 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f8c0a8e86f04f32b79743f819948544 with delete_type TABLET_DATA_TOMBSTONED (TS a29cd590999b4f32849ff5689cd53c11 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54036
I20240501 08:12:43.705482 1300 consensus_queue.cc:1035] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:43.708230 939 raft_consensus.cc:2932] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:43.709023 30803 raft_consensus.cc:2932] T 893c1555ca0849da8f48a4f253b42153 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:43.716728 28151 raft_consensus.cc:2932] T 893c1555ca0849da8f48a4f253b42153 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:43.716835 27612 catalog_manager.cc:5484] T 893c1555ca0849da8f48a4f253b42153 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:43.731385 1475 tablet_replica.cc:330] T 4f8c0a8e86f04f32b79743f819948544 P a29cd590999b4f32849ff5689cd53c11: stopping tablet replica
I20240501 08:12:43.731740 1475 raft_consensus.cc:2229] T 4f8c0a8e86f04f32b79743f819948544 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:43.732065 1475 raft_consensus.cc:2258] T 4f8c0a8e86f04f32b79743f819948544 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:43.733023 1475 ts_tablet_manager.cc:1875] T 4f8c0a8e86f04f32b79743f819948544 P a29cd590999b4f32849ff5689cd53c11: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:43.736821 28019 raft_consensus.cc:2932] T 893c1555ca0849da8f48a4f253b42153 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:43.740912 1475 ts_tablet_manager.cc:1888] T 4f8c0a8e86f04f32b79743f819948544 P a29cd590999b4f32849ff5689cd53c11: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:43.741134 1475 log.cc:1196] T 4f8c0a8e86f04f32b79743f819948544 P a29cd590999b4f32849ff5689cd53c11: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/wals/4f8c0a8e86f04f32b79743f819948544
I20240501 08:12:43.741185 30536 consensus_queue.cc:237] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } }
I20240501 08:12:43.743953 28149 raft_consensus.cc:1261] T 893c1555ca0849da8f48a4f253b42153 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:43.744737 1473 consensus_queue.cc:1035] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:43.747462 28019 raft_consensus.cc:1261] T 893c1555ca0849da8f48a4f253b42153 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:43.747555 27606 catalog_manager.cc:4830] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): tablet 4f8c0a8e86f04f32b79743f819948544 (table rebalancer_extra_table_44 [id=a727b21f159445ba8749f9bf1bec2b3d]) successfully deleted
I20240501 08:12:43.747334 1473 raft_consensus.cc:2932] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:43.748224 1300 consensus_queue.cc:1035] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:43.750591 28149 raft_consensus.cc:2932] T 893c1555ca0849da8f48a4f253b42153 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:43.751292 28019 raft_consensus.cc:2932] T 893c1555ca0849da8f48a4f253b42153 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:43.755481 27607 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 893c1555ca0849da8f48a4f253b42153 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:43.758370 27612 catalog_manager.cc:5484] T 893c1555ca0849da8f48a4f253b42153 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: config changed from index 3 to 4, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:43.769490 30784 tablet_service.cc:1514] Processing DeleteTablet for tablet 893c1555ca0849da8f48a4f253b42153 with delete_type TABLET_DATA_TOMBSTONED (TS 64998323a1f4482d8704bf20594d13a5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38952
I20240501 08:12:43.779585 1487 tablet_replica.cc:330] T 893c1555ca0849da8f48a4f253b42153 P 64998323a1f4482d8704bf20594d13a5: stopping tablet replica
I20240501 08:12:43.779831 1487 raft_consensus.cc:2229] T 893c1555ca0849da8f48a4f253b42153 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:43.780071 1487 raft_consensus.cc:2258] T 893c1555ca0849da8f48a4f253b42153 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:43.780822 1487 ts_tablet_manager.cc:1875] T 893c1555ca0849da8f48a4f253b42153 P 64998323a1f4482d8704bf20594d13a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:43.787061 1487 ts_tablet_manager.cc:1888] T 893c1555ca0849da8f48a4f253b42153 P 64998323a1f4482d8704bf20594d13a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:43.787231 1487 log.cc:1196] T 893c1555ca0849da8f48a4f253b42153 P 64998323a1f4482d8704bf20594d13a5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal/wals/893c1555ca0849da8f48a4f253b42153
I20240501 08:12:43.787984 27605 catalog_manager.cc:4830] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): tablet 893c1555ca0849da8f48a4f253b42153 (table rebalancer_extra_table_36 [id=a0611396e39b4372a1b4597945f4a169]) successfully deleted
I20240501 08:12:43.855397 27890 consensus_queue.cc:237] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
I20240501 08:12:43.857650 27761 raft_consensus.cc:1261] T 7f7211c8c0e041bf89930aa50a4438ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:12:43.858086 27828 consensus_peers.cc:485] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af -> Peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): Couldn't send request to peer 31811708c6774037b82df44024b229b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f7211c8c0e041bf89930aa50a4438ea. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:43.858348 1430 consensus_queue.cc:1035] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:43.858623 28020 raft_consensus.cc:1261] T 7f7211c8c0e041bf89930aa50a4438ea P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:43.859179 1430 consensus_queue.cc:1035] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:43.863253 1419 raft_consensus.cc:2932] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:12:43.864025 28020 raft_consensus.cc:2932] T 7f7211c8c0e041bf89930aa50a4438ea P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:12:43.865393 27761 raft_consensus.cc:2932] T 7f7211c8c0e041bf89930aa50a4438ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:12:43.870254 27612 catalog_manager.cc:5484] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af reported cstate change: config changed from index -1 to 4, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:43.891323 27890 consensus_queue.cc:237] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
W20240501 08:12:43.894058 27825 consensus_peers.cc:485] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af -> Peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Couldn't send request to peer 9a022e1a501f4d44a0468054d6c3de47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9820951bffc749139229e2b79c8ab597. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:43.894094 28020 raft_consensus.cc:1261] T 9820951bffc749139229e2b79c8ab597 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:43.894212 27758 raft_consensus.cc:1261] T 9820951bffc749139229e2b79c8ab597 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:43.894748 1419 consensus_queue.cc:1035] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:43.895305 1430 consensus_queue.cc:1035] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:43.901679 1419 raft_consensus.cc:2932] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:43.902568 28020 raft_consensus.cc:2932] T 9820951bffc749139229e2b79c8ab597 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:43.905921 27758 raft_consensus.cc:2932] T 9820951bffc749139229e2b79c8ab597 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:43.912830 27612 catalog_manager.cc:5484] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af reported cstate change: config changed from index -1 to 4, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:43.959760 27758 consensus_queue.cc:237] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
I20240501 08:12:43.962139 27890 raft_consensus.cc:1261] T e3d07789edc74dec975e35e9d94cb0ef P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:43.962710 312 consensus_queue.cc:1035] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:43.962815 28149 raft_consensus.cc:1261] T e3d07789edc74dec975e35e9d94cb0ef P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:12:43.963053 27695 consensus_peers.cc:485] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 -> Peer 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): Couldn't send request to peer 64998323a1f4482d8704bf20594d13a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e3d07789edc74dec975e35e9d94cb0ef. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:43.963369 312 consensus_queue.cc:1035] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:43.965489 1432 raft_consensus.cc:2932] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:12:43.966172 27890 raft_consensus.cc:2932] T e3d07789edc74dec975e35e9d94cb0ef P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:12:43.967756 28149 raft_consensus.cc:2932] T e3d07789edc74dec975e35e9d94cb0ef P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:12:43.973580 27616 catalog_manager.cc:5484] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index -1 to 4, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:44.002517 30935 consensus_queue.cc:237] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:44.004892 27758 raft_consensus.cc:1261] T c0e30c62c2d441618195f30f15647487 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:44.005034 30669 raft_consensus.cc:1261] T c0e30c62c2d441618195f30f15647487 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:44.005784 1378 consensus_queue.cc:1035] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:44.006383 1379 consensus_queue.cc:1035] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:12:44.006727 30873 consensus_peers.cc:485] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0e30c62c2d441618195f30f15647487. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:44.009567 1543 raft_consensus.cc:2932] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:44.010241 27758 raft_consensus.cc:2932] T c0e30c62c2d441618195f30f15647487 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:44.010381 30669 raft_consensus.cc:2932] T c0e30c62c2d441618195f30f15647487 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:44.017989 27612 catalog_manager.cc:5484] T c0e30c62c2d441618195f30f15647487 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:44.038971 27758 consensus_queue.cc:237] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
I20240501 08:12:44.041488 27890 raft_consensus.cc:1261] T 0e0f87fa9df24b43bedf063a9063bcc5 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:44.041492 28019 raft_consensus.cc:1261] T 0e0f87fa9df24b43bedf063a9063bcc5 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:44.042069 1432 consensus_queue.cc:1035] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:44.042388 1426 consensus_queue.cc:1035] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:44.045593 1432 raft_consensus.cc:2932] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:12:44.046439 27890 raft_consensus.cc:2932] T 0e0f87fa9df24b43bedf063a9063bcc5 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
W20240501 08:12:44.047835 27697 consensus_peers.cc:485] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 -> Peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): Couldn't send request to peer 31811708c6774037b82df44024b229b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e0f87fa9df24b43bedf063a9063bcc5. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:44.054387 27618 catalog_manager.cc:5484] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index -1 to 4, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:44.078563 28019 raft_consensus.cc:2932] T 0e0f87fa9df24b43bedf063a9063bcc5 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:12:44.144551 28149 consensus_queue.cc:237] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:12:44.147388 27758 raft_consensus.cc:1261] T 132d1b9866574dcea3beed5835e4a22f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:44.147955 28020 raft_consensus.cc:1261] T 132d1b9866574dcea3beed5835e4a22f P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:12:44.148821 28085 consensus_peers.cc:485] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6 -> Peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Couldn't send request to peer 9a022e1a501f4d44a0468054d6c3de47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 132d1b9866574dcea3beed5835e4a22f. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:44.150046 1435 consensus_queue.cc:1035] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:44.150377 840 consensus_queue.cc:1035] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:44.153692 1199 raft_consensus.cc:2932] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:44.159045 27758 raft_consensus.cc:2932] T 132d1b9866574dcea3beed5835e4a22f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:44.160763 27612 catalog_manager.cc:5484] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: config changed from index -1 to 4, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:44.174010 28020 raft_consensus.cc:2932] T 132d1b9866574dcea3beed5835e4a22f P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:44.202039 28149 consensus_queue.cc:237] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
I20240501 08:12:44.204903 30935 raft_consensus.cc:1261] T 98c8e3a6f66a42b08507668f7c3a8af1 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:44.205238 27891 raft_consensus.cc:1261] T 98c8e3a6f66a42b08507668f7c3a8af1 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:44.205480 840 consensus_queue.cc:1035] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:12:44.205909 28086 consensus_peers.cc:485] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 -> Peer 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): Couldn't send request to peer 64998323a1f4482d8704bf20594d13a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98c8e3a6f66a42b08507668f7c3a8af1. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:44.206102 840 consensus_queue.cc:1035] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:44.209784 1435 raft_consensus.cc:2932] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:12:44.210477 30935 raft_consensus.cc:2932] T 98c8e3a6f66a42b08507668f7c3a8af1 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:12:44.210819 27890 raft_consensus.cc:2932] T 98c8e3a6f66a42b08507668f7c3a8af1 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:12:44.217646 27618 catalog_manager.cc:5484] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: config changed from index -1 to 2, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:44.236436 30935 consensus_queue.cc:237] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:44.239686 30536 raft_consensus.cc:1261] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:12:44.239996 30873 consensus_peers.cc:485] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ecab521101074a7b80b862642059daf2. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:44.240469 27758 raft_consensus.cc:1261] T ecab521101074a7b80b862642059daf2 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:44.240953 1379 consensus_queue.cc:1035] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:44.241636 1379 consensus_queue.cc:1035] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:44.244632 1543 raft_consensus.cc:2932] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:44.246845 27761 raft_consensus.cc:2932] T ecab521101074a7b80b862642059daf2 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:44.250237 30537 raft_consensus.cc:2932] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:44.253983 27618 catalog_manager.cc:5484] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:44.274698 1582 ts_tablet_manager.cc:902] T 7f7211c8c0e041bf89930aa50a4438ea P 31811708c6774037b82df44024b229b6: Initiating tablet copy from peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:12:44.284184 1582 tablet_copy_client.cc:308] T 7f7211c8c0e041bf89930aa50a4438ea P 31811708c6774037b82df44024b229b6: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.130:36503
I20240501 08:12:44.284651 27761 consensus_queue.cc:237] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
I20240501 08:12:44.284897 27912 tablet_copy_service.cc:140] P f1525cbc82a24d13b65f136695c903af: Received BeginTabletCopySession request for tablet 7f7211c8c0e041bf89930aa50a4438ea from peer 31811708c6774037b82df44024b229b6 ({username='slave'} at 127.21.84.133:34301)
I20240501 08:12:44.285032 27912 tablet_copy_service.cc:161] P f1525cbc82a24d13b65f136695c903af: Beginning new tablet copy session on tablet 7f7211c8c0e041bf89930aa50a4438ea from peer 31811708c6774037b82df44024b229b6 at {username='slave'} at 127.21.84.133:34301: session id = 31811708c6774037b82df44024b229b6-7f7211c8c0e041bf89930aa50a4438ea
I20240501 08:12:44.287333 28020 raft_consensus.cc:1261] T ac4914417d9e44e7a9591be5386f1d8b P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:12:44.287526 27697 consensus_peers.cc:485] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43 -> Peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): Couldn't send request to peer 31811708c6774037b82df44024b229b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac4914417d9e44e7a9591be5386f1d8b. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:44.287899 312 consensus_queue.cc:1035] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:44.288121 27912 tablet_copy_source_session.cc:205] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:44.289034 27890 raft_consensus.cc:1261] T ac4914417d9e44e7a9591be5386f1d8b P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:44.289549 1582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f7211c8c0e041bf89930aa50a4438ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:44.291464 312 consensus_queue.cc:1035] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:44.300850 1582 tablet_copy_client.cc:783] T 7f7211c8c0e041bf89930aa50a4438ea P 31811708c6774037b82df44024b229b6: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:44.301173 1582 tablet_copy_client.cc:647] T 7f7211c8c0e041bf89930aa50a4438ea P 31811708c6774037b82df44024b229b6: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:44.306213 1582 tablet_copy_client.cc:523] T 7f7211c8c0e041bf89930aa50a4438ea P 31811708c6774037b82df44024b229b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:44.305888 312 raft_consensus.cc:2932] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:12:44.307113 27890 raft_consensus.cc:2932] T ac4914417d9e44e7a9591be5386f1d8b P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:12:44.310429 1582 tablet_bootstrap.cc:492] T 7f7211c8c0e041bf89930aa50a4438ea P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:44.315322 27612 catalog_manager.cc:5484] T ac4914417d9e44e7a9591be5386f1d8b P f1525cbc82a24d13b65f136695c903af reported cstate change: config changed from index -1 to 4, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:12:44.322526 28018 raft_consensus.cc:2932] T ac4914417d9e44e7a9591be5386f1d8b P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
W20240501 08:12:44.339031 30737 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.132:41323, user_credentials={real_user=slave}} blocked reactor thread for 43874.5us
I20240501 08:12:44.378072 28020 consensus_queue.cc:237] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:12:44.379962 1582 tablet_bootstrap.cc:492] T 7f7211c8c0e041bf89930aa50a4438ea P 31811708c6774037b82df44024b229b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:44.380424 1582 tablet_bootstrap.cc:492] T 7f7211c8c0e041bf89930aa50a4438ea P 31811708c6774037b82df44024b229b6: Bootstrap complete.
I20240501 08:12:44.380755 1582 ts_tablet_manager.cc:1367] T 7f7211c8c0e041bf89930aa50a4438ea P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.070s user 0.051s sys 0.007s
I20240501 08:12:44.381562 1582 raft_consensus.cc:348] T 7f7211c8c0e041bf89930aa50a4438ea P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
I20240501 08:12:44.381842 1582 raft_consensus.cc:729] T 7f7211c8c0e041bf89930aa50a4438ea P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: LEARNER
I20240501 08:12:44.382098 1582 consensus_queue.cc:260] T 7f7211c8c0e041bf89930aa50a4438ea P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
I20240501 08:12:44.382746 1582 ts_tablet_manager.cc:1398] T 7f7211c8c0e041bf89930aa50a4438ea P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:12:44.384752 27912 tablet_copy_service.cc:342] P f1525cbc82a24d13b65f136695c903af: Request end of tablet copy session 31811708c6774037b82df44024b229b6-7f7211c8c0e041bf89930aa50a4438ea received from {username='slave'} at 127.21.84.133:34301
W20240501 08:12:44.384786 27955 consensus_peers.cc:485] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 -> Peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Couldn't send request to peer 9a022e1a501f4d44a0468054d6c3de47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6fb30f10c754d3a8e73e124ec59768b. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:44.384876 27912 tablet_copy_service.cc:433] P f1525cbc82a24d13b65f136695c903af: ending tablet copy session 31811708c6774037b82df44024b229b6-7f7211c8c0e041bf89930aa50a4438ea on tablet 7f7211c8c0e041bf89930aa50a4438ea with peer 31811708c6774037b82df44024b229b6
I20240501 08:12:44.384923 28149 raft_consensus.cc:1261] T e6fb30f10c754d3a8e73e124ec59768b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:44.385648 877 consensus_queue.cc:1035] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:44.387497 27761 raft_consensus.cc:1261] T e6fb30f10c754d3a8e73e124ec59768b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:44.388100 877 consensus_queue.cc:1035] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:44.407356 877 raft_consensus.cc:2932] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:44.411943 27758 raft_consensus.cc:2932] T e6fb30f10c754d3a8e73e124ec59768b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:44.412007 28152 raft_consensus.cc:2932] T e6fb30f10c754d3a8e73e124ec59768b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:44.417657 27612 catalog_manager.cc:5484] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index -1 to 4, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:44.441032 1609 ts_tablet_manager.cc:902] T 9820951bffc749139229e2b79c8ab597 P 9a022e1a501f4d44a0468054d6c3de47: Initiating tablet copy from peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:12:44.446022 1609 tablet_copy_client.cc:308] T 9820951bffc749139229e2b79c8ab597 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.130:36503
I20240501 08:12:44.469662 28149 consensus_queue.cc:237] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
I20240501 08:12:44.474887 27912 tablet_copy_service.cc:140] P f1525cbc82a24d13b65f136695c903af: Received BeginTabletCopySession request for tablet 9820951bffc749139229e2b79c8ab597 from peer 9a022e1a501f4d44a0468054d6c3de47 ({username='slave'} at 127.21.84.134:39531)
I20240501 08:12:44.475098 27912 tablet_copy_service.cc:161] P f1525cbc82a24d13b65f136695c903af: Beginning new tablet copy session on tablet 9820951bffc749139229e2b79c8ab597 from peer 9a022e1a501f4d44a0468054d6c3de47 at {username='slave'} at 127.21.84.134:39531: session id = 9a022e1a501f4d44a0468054d6c3de47-9820951bffc749139229e2b79c8ab597
W20240501 08:12:44.476989 28086 consensus_peers.cc:485] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 -> Peer 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): Couldn't send request to peer 64998323a1f4482d8704bf20594d13a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30130709805e4fa6ae94a46034710863. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:44.478091 27912 tablet_copy_source_session.cc:205] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:44.478219 27758 raft_consensus.cc:1261] T 30130709805e4fa6ae94a46034710863 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:44.478236 28020 raft_consensus.cc:1261] T 30130709805e4fa6ae94a46034710863 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:44.478971 1435 consensus_queue.cc:1035] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:44.479307 1199 consensus_queue.cc:1035] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:44.480037 1609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9820951bffc749139229e2b79c8ab597. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:44.489456 1609 tablet_copy_client.cc:783] T 9820951bffc749139229e2b79c8ab597 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:44.489818 1609 tablet_copy_client.cc:647] T 9820951bffc749139229e2b79c8ab597 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:44.492288 1609 tablet_copy_client.cc:523] T 9820951bffc749139229e2b79c8ab597 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:44.494124 1619 ts_tablet_manager.cc:902] T c0e30c62c2d441618195f30f15647487 P f1525cbc82a24d13b65f136695c903af: Initiating tablet copy from peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:44.497404 1609 tablet_bootstrap.cc:492] T 9820951bffc749139229e2b79c8ab597 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:44.525326 1199 raft_consensus.cc:2932] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:12:44.526134 28018 raft_consensus.cc:2932] T 30130709805e4fa6ae94a46034710863 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:12:44.534718 27618 catalog_manager.cc:5484] T 30130709805e4fa6ae94a46034710863 P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index -1 to 4, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:12:44.542596 27758 raft_consensus.cc:2932] T 30130709805e4fa6ae94a46034710863 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:12:44.573359 1623 ts_tablet_manager.cc:902] T 132d1b9866574dcea3beed5835e4a22f P 9a022e1a501f4d44a0468054d6c3de47: Initiating tablet copy from peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:44.574965 1621 ts_tablet_manager.cc:902] T e3d07789edc74dec975e35e9d94cb0ef P 64998323a1f4482d8704bf20594d13a5: Initiating tablet copy from peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:44.574920 1619 tablet_copy_client.cc:308] T c0e30c62c2d441618195f30f15647487 P f1525cbc82a24d13b65f136695c903af: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.136:46845
I20240501 08:12:44.575780 30957 tablet_copy_service.cc:140] P a29cd590999b4f32849ff5689cd53c11: Received BeginTabletCopySession request for tablet c0e30c62c2d441618195f30f15647487 from peer f1525cbc82a24d13b65f136695c903af ({username='slave'} at 127.21.84.130:49339)
I20240501 08:12:44.575979 30957 tablet_copy_service.cc:161] P a29cd590999b4f32849ff5689cd53c11: Beginning new tablet copy session on tablet c0e30c62c2d441618195f30f15647487 from peer f1525cbc82a24d13b65f136695c903af at {username='slave'} at 127.21.84.130:49339: session id = f1525cbc82a24d13b65f136695c903af-c0e30c62c2d441618195f30f15647487
I20240501 08:12:44.579319 30957 tablet_copy_source_session.cc:205] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:44.581799 1619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0e30c62c2d441618195f30f15647487. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:44.590925 1619 tablet_copy_client.cc:783] T c0e30c62c2d441618195f30f15647487 P f1525cbc82a24d13b65f136695c903af: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:44.591224 1619 tablet_copy_client.cc:647] T c0e30c62c2d441618195f30f15647487 P f1525cbc82a24d13b65f136695c903af: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:44.593559 1623 tablet_copy_client.cc:308] T 132d1b9866574dcea3beed5835e4a22f P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.132:41323
I20240501 08:12:44.594143 1619 tablet_copy_client.cc:523] T c0e30c62c2d441618195f30f15647487 P f1525cbc82a24d13b65f136695c903af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:44.594377 28172 tablet_copy_service.cc:140] P 91d11eae5ef6432ca86eb8ed446b6ea6: Received BeginTabletCopySession request for tablet 132d1b9866574dcea3beed5835e4a22f from peer 9a022e1a501f4d44a0468054d6c3de47 ({username='slave'} at 127.21.84.134:40845)
I20240501 08:12:44.594523 28172 tablet_copy_service.cc:161] P 91d11eae5ef6432ca86eb8ed446b6ea6: Beginning new tablet copy session on tablet 132d1b9866574dcea3beed5835e4a22f from peer 9a022e1a501f4d44a0468054d6c3de47 at {username='slave'} at 127.21.84.134:40845: session id = 9a022e1a501f4d44a0468054d6c3de47-132d1b9866574dcea3beed5835e4a22f
I20240501 08:12:44.598762 1621 tablet_copy_client.cc:308] T e3d07789edc74dec975e35e9d94cb0ef P 64998323a1f4482d8704bf20594d13a5: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.129:43471
I20240501 08:12:44.599735 27781 tablet_copy_service.cc:140] P 6331f77dae9d452184e8cd0cbf5f1b43: Received BeginTabletCopySession request for tablet e3d07789edc74dec975e35e9d94cb0ef from peer 64998323a1f4482d8704bf20594d13a5 ({username='slave'} at 127.21.84.135:52115)
I20240501 08:12:44.599916 27781 tablet_copy_service.cc:161] P 6331f77dae9d452184e8cd0cbf5f1b43: Beginning new tablet copy session on tablet e3d07789edc74dec975e35e9d94cb0ef from peer 64998323a1f4482d8704bf20594d13a5 at {username='slave'} at 127.21.84.135:52115: session id = 64998323a1f4482d8704bf20594d13a5-e3d07789edc74dec975e35e9d94cb0ef
I20240501 08:12:44.600251 28172 tablet_copy_source_session.cc:205] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:44.601037 1619 tablet_bootstrap.cc:492] T c0e30c62c2d441618195f30f15647487 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:44.614111 1619 tablet_bootstrap.cc:492] T c0e30c62c2d441618195f30f15647487 P f1525cbc82a24d13b65f136695c903af: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:44.614579 1619 tablet_bootstrap.cc:492] T c0e30c62c2d441618195f30f15647487 P f1525cbc82a24d13b65f136695c903af: Bootstrap complete.
I20240501 08:12:44.614852 1619 ts_tablet_manager.cc:1367] T c0e30c62c2d441618195f30f15647487 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.011s
I20240501 08:12:44.615633 1619 raft_consensus.cc:348] T c0e30c62c2d441618195f30f15647487 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:44.615936 1619 raft_consensus.cc:729] T c0e30c62c2d441618195f30f15647487 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: LEARNER
I20240501 08:12:44.616154 1619 consensus_queue.cc:260] T c0e30c62c2d441618195f30f15647487 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:44.616833 27781 tablet_copy_source_session.cc:205] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:44.618264 1623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 132d1b9866574dcea3beed5835e4a22f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:44.618247 1619 ts_tablet_manager.cc:1398] T c0e30c62c2d441618195f30f15647487 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:44.620489 30957 tablet_copy_service.cc:342] P a29cd590999b4f32849ff5689cd53c11: Request end of tablet copy session f1525cbc82a24d13b65f136695c903af-c0e30c62c2d441618195f30f15647487 received from {username='slave'} at 127.21.84.130:49339
I20240501 08:12:44.620637 30957 tablet_copy_service.cc:433] P a29cd590999b4f32849ff5689cd53c11: ending tablet copy session f1525cbc82a24d13b65f136695c903af-c0e30c62c2d441618195f30f15647487 on tablet c0e30c62c2d441618195f30f15647487 with peer f1525cbc82a24d13b65f136695c903af
I20240501 08:12:44.625355 1621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3d07789edc74dec975e35e9d94cb0ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:44.634037 1621 tablet_copy_client.cc:783] T e3d07789edc74dec975e35e9d94cb0ef P 64998323a1f4482d8704bf20594d13a5: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:44.634346 1621 tablet_copy_client.cc:647] T e3d07789edc74dec975e35e9d94cb0ef P 64998323a1f4482d8704bf20594d13a5: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:44.636076 1621 tablet_copy_client.cc:523] T e3d07789edc74dec975e35e9d94cb0ef P 64998323a1f4482d8704bf20594d13a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:44.637068 1623 tablet_copy_client.cc:783] T 132d1b9866574dcea3beed5835e4a22f P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:44.637521 1623 tablet_copy_client.cc:647] T 132d1b9866574dcea3beed5835e4a22f P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:44.640158 1621 tablet_bootstrap.cc:492] T e3d07789edc74dec975e35e9d94cb0ef P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:44.640413 1623 tablet_copy_client.cc:523] T 132d1b9866574dcea3beed5835e4a22f P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:44.646107 1623 tablet_bootstrap.cc:492] T 132d1b9866574dcea3beed5835e4a22f P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:44.661026 1582 ts_tablet_manager.cc:902] T 0e0f87fa9df24b43bedf063a9063bcc5 P 31811708c6774037b82df44024b229b6: Initiating tablet copy from peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:44.675136 1641 ts_tablet_manager.cc:902] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5: Initiating tablet copy from peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:44.686759 30934 consensus_queue.cc:237] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:44.690789 30669 raft_consensus.cc:1261] T b605254185d140d8ab90fcaa5229f168 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:12:44.691017 30873 consensus_peers.cc:485] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b605254185d140d8ab90fcaa5229f168. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:44.692032 1379 consensus_queue.cc:1035] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:44.694262 28152 raft_consensus.cc:1261] T b605254185d140d8ab90fcaa5229f168 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:44.694819 1379 consensus_queue.cc:1035] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:44.702805 1582 tablet_copy_client.cc:308] T 0e0f87fa9df24b43bedf063a9063bcc5 P 31811708c6774037b82df44024b229b6: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.129:43471
I20240501 08:12:44.703614 27781 tablet_copy_service.cc:140] P 6331f77dae9d452184e8cd0cbf5f1b43: Received BeginTabletCopySession request for tablet 0e0f87fa9df24b43bedf063a9063bcc5 from peer 31811708c6774037b82df44024b229b6 ({username='slave'} at 127.21.84.133:35347)
I20240501 08:12:44.703760 27781 tablet_copy_service.cc:161] P 6331f77dae9d452184e8cd0cbf5f1b43: Beginning new tablet copy session on tablet 0e0f87fa9df24b43bedf063a9063bcc5 from peer 31811708c6774037b82df44024b229b6 at {username='slave'} at 127.21.84.133:35347: session id = 31811708c6774037b82df44024b229b6-0e0f87fa9df24b43bedf063a9063bcc5
I20240501 08:12:44.709031 27781 tablet_copy_source_session.cc:205] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:44.710563 1582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e0f87fa9df24b43bedf063a9063bcc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:44.718290 1582 tablet_copy_client.cc:783] T 0e0f87fa9df24b43bedf063a9063bcc5 P 31811708c6774037b82df44024b229b6: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:44.718585 1582 tablet_copy_client.cc:647] T 0e0f87fa9df24b43bedf063a9063bcc5 P 31811708c6774037b82df44024b229b6: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:44.720415 1582 tablet_copy_client.cc:523] T 0e0f87fa9df24b43bedf063a9063bcc5 P 31811708c6774037b82df44024b229b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:44.740789 1582 tablet_bootstrap.cc:492] T 0e0f87fa9df24b43bedf063a9063bcc5 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:44.741045 30536 raft_consensus.cc:1203] T 7f7211c8c0e041bf89930aa50a4438ea P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:44.762485 1623 tablet_bootstrap.cc:492] T 132d1b9866574dcea3beed5835e4a22f P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:44.763118 1623 tablet_bootstrap.cc:492] T 132d1b9866574dcea3beed5835e4a22f P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap complete.
I20240501 08:12:44.763509 1623 ts_tablet_manager.cc:1367] T 132d1b9866574dcea3beed5835e4a22f P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.118s user 0.058s sys 0.033s
I20240501 08:12:44.764351 1623 raft_consensus.cc:348] T 132d1b9866574dcea3beed5835e4a22f P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:12:44.764699 1623 raft_consensus.cc:729] T 132d1b9866574dcea3beed5835e4a22f P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: LEARNER
I20240501 08:12:44.764981 1623 consensus_queue.cc:260] T 132d1b9866574dcea3beed5835e4a22f P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:12:44.766276 1609 tablet_bootstrap.cc:492] T 9820951bffc749139229e2b79c8ab597 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:44.766770 1609 tablet_bootstrap.cc:492] T 9820951bffc749139229e2b79c8ab597 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap complete.
I20240501 08:12:44.767091 1609 ts_tablet_manager.cc:1367] T 9820951bffc749139229e2b79c8ab597 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.270s user 0.051s sys 0.008s
I20240501 08:12:44.767551 1623 ts_tablet_manager.cc:1398] T 132d1b9866574dcea3beed5835e4a22f P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 08:12:44.768198 28172 tablet_copy_service.cc:342] P 91d11eae5ef6432ca86eb8ed446b6ea6: Request end of tablet copy session 9a022e1a501f4d44a0468054d6c3de47-132d1b9866574dcea3beed5835e4a22f received from {username='slave'} at 127.21.84.134:40845
I20240501 08:12:44.768306 28172 tablet_copy_service.cc:433] P 91d11eae5ef6432ca86eb8ed446b6ea6: ending tablet copy session 9a022e1a501f4d44a0468054d6c3de47-132d1b9866574dcea3beed5835e4a22f on tablet 132d1b9866574dcea3beed5835e4a22f with peer 9a022e1a501f4d44a0468054d6c3de47
I20240501 08:12:44.774857 1609 raft_consensus.cc:348] T 9820951bffc749139229e2b79c8ab597 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:12:44.775206 1609 raft_consensus.cc:729] T 9820951bffc749139229e2b79c8ab597 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: LEARNER
I20240501 08:12:44.775468 1609 consensus_queue.cc:260] T 9820951bffc749139229e2b79c8ab597 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:12:44.778970 1609 ts_tablet_manager.cc:1398] T 9820951bffc749139229e2b79c8ab597 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20240501 08:12:44.779603 27912 tablet_copy_service.cc:342] P f1525cbc82a24d13b65f136695c903af: Request end of tablet copy session 9a022e1a501f4d44a0468054d6c3de47-9820951bffc749139229e2b79c8ab597 received from {username='slave'} at 127.21.84.134:39531
I20240501 08:12:44.779747 27912 tablet_copy_service.cc:433] P f1525cbc82a24d13b65f136695c903af: ending tablet copy session 9a022e1a501f4d44a0468054d6c3de47-9820951bffc749139229e2b79c8ab597 on tablet 9820951bffc749139229e2b79c8ab597 with peer 9a022e1a501f4d44a0468054d6c3de47
I20240501 08:12:44.787003 1623 ts_tablet_manager.cc:902] T e6fb30f10c754d3a8e73e124ec59768b P 9a022e1a501f4d44a0468054d6c3de47: Initiating tablet copy from peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:44.808259 1379 raft_consensus.cc:2932] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:44.819482 28149 raft_consensus.cc:2932] T b605254185d140d8ab90fcaa5229f168 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:44.829890 1621 tablet_bootstrap.cc:492] T e3d07789edc74dec975e35e9d94cb0ef P 64998323a1f4482d8704bf20594d13a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:44.830469 1621 tablet_bootstrap.cc:492] T e3d07789edc74dec975e35e9d94cb0ef P 64998323a1f4482d8704bf20594d13a5: Bootstrap complete.
I20240501 08:12:44.830849 1621 ts_tablet_manager.cc:1367] T e3d07789edc74dec975e35e9d94cb0ef P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.191s user 0.054s sys 0.007s
I20240501 08:12:44.831853 1621 raft_consensus.cc:348] T e3d07789edc74dec975e35e9d94cb0ef P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
I20240501 08:12:44.832289 1621 raft_consensus.cc:729] T e3d07789edc74dec975e35e9d94cb0ef P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: LEARNER
I20240501 08:12:44.832489 1621 consensus_queue.cc:260] T e3d07789edc74dec975e35e9d94cb0ef P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
I20240501 08:12:44.835088 27612 catalog_manager.cc:5484] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:44.839713 1619 ts_tablet_manager.cc:902] T ecab521101074a7b80b862642059daf2 P f1525cbc82a24d13b65f136695c903af: Initiating tablet copy from peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:44.852377 1621 ts_tablet_manager.cc:1398] T e3d07789edc74dec975e35e9d94cb0ef P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.016s user 0.002s sys 0.001s
I20240501 08:12:44.853178 27781 tablet_copy_service.cc:342] P 6331f77dae9d452184e8cd0cbf5f1b43: Request end of tablet copy session 64998323a1f4482d8704bf20594d13a5-e3d07789edc74dec975e35e9d94cb0ef received from {username='slave'} at 127.21.84.135:52115
I20240501 08:12:44.853299 27781 tablet_copy_service.cc:433] P 6331f77dae9d452184e8cd0cbf5f1b43: ending tablet copy session 64998323a1f4482d8704bf20594d13a5-e3d07789edc74dec975e35e9d94cb0ef on tablet e3d07789edc74dec975e35e9d94cb0ef with peer 64998323a1f4482d8704bf20594d13a5
I20240501 08:12:44.854144 1641 tablet_copy_client.cc:308] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.132:41323
I20240501 08:12:44.853914 30672 raft_consensus.cc:2932] T b605254185d140d8ab90fcaa5229f168 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:44.858362 28172 tablet_copy_service.cc:140] P 91d11eae5ef6432ca86eb8ed446b6ea6: Received BeginTabletCopySession request for tablet 98c8e3a6f66a42b08507668f7c3a8af1 from peer 64998323a1f4482d8704bf20594d13a5 ({username='slave'} at 127.21.84.135:48917)
I20240501 08:12:44.858589 28172 tablet_copy_service.cc:161] P 91d11eae5ef6432ca86eb8ed446b6ea6: Beginning new tablet copy session on tablet 98c8e3a6f66a42b08507668f7c3a8af1 from peer 64998323a1f4482d8704bf20594d13a5 at {username='slave'} at 127.21.84.135:48917: session id = 64998323a1f4482d8704bf20594d13a5-98c8e3a6f66a42b08507668f7c3a8af1
I20240501 08:12:44.870523 28172 tablet_copy_source_session.cc:205] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:44.873150 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98c8e3a6f66a42b08507668f7c3a8af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:44.890688 1641 tablet_copy_client.cc:783] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:44.891130 1641 tablet_copy_client.cc:647] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:44.893296 1619 tablet_copy_client.cc:308] T ecab521101074a7b80b862642059daf2 P f1525cbc82a24d13b65f136695c903af: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.136:46845
I20240501 08:12:44.893394 1641 tablet_copy_client.cc:523] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:44.894079 30957 tablet_copy_service.cc:140] P a29cd590999b4f32849ff5689cd53c11: Received BeginTabletCopySession request for tablet ecab521101074a7b80b862642059daf2 from peer f1525cbc82a24d13b65f136695c903af ({username='slave'} at 127.21.84.130:49339)
I20240501 08:12:44.894189 1582 tablet_bootstrap.cc:492] T 0e0f87fa9df24b43bedf063a9063bcc5 P 31811708c6774037b82df44024b229b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:44.894272 30957 tablet_copy_service.cc:161] P a29cd590999b4f32849ff5689cd53c11: Beginning new tablet copy session on tablet ecab521101074a7b80b862642059daf2 from peer f1525cbc82a24d13b65f136695c903af at {username='slave'} at 127.21.84.130:49339: session id = f1525cbc82a24d13b65f136695c903af-ecab521101074a7b80b862642059daf2
I20240501 08:12:44.894730 1582 tablet_bootstrap.cc:492] T 0e0f87fa9df24b43bedf063a9063bcc5 P 31811708c6774037b82df44024b229b6: Bootstrap complete.
I20240501 08:12:44.895015 1582 ts_tablet_manager.cc:1367] T 0e0f87fa9df24b43bedf063a9063bcc5 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.154s user 0.053s sys 0.001s
I20240501 08:12:44.895831 1582 raft_consensus.cc:348] T 0e0f87fa9df24b43bedf063a9063bcc5 P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
I20240501 08:12:44.896099 1582 raft_consensus.cc:729] T 0e0f87fa9df24b43bedf063a9063bcc5 P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: LEARNER
I20240501 08:12:44.896354 1582 consensus_queue.cc:260] T 0e0f87fa9df24b43bedf063a9063bcc5 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
I20240501 08:12:44.896943 1582 ts_tablet_manager.cc:1398] T 0e0f87fa9df24b43bedf063a9063bcc5 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:12:44.897615 30957 tablet_copy_source_session.cc:205] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:44.897677 27781 tablet_copy_service.cc:342] P 6331f77dae9d452184e8cd0cbf5f1b43: Request end of tablet copy session 31811708c6774037b82df44024b229b6-0e0f87fa9df24b43bedf063a9063bcc5 received from {username='slave'} at 127.21.84.133:35347
I20240501 08:12:44.897781 27781 tablet_copy_service.cc:433] P 6331f77dae9d452184e8cd0cbf5f1b43: ending tablet copy session 31811708c6774037b82df44024b229b6-0e0f87fa9df24b43bedf063a9063bcc5 on tablet 0e0f87fa9df24b43bedf063a9063bcc5 with peer 31811708c6774037b82df44024b229b6
I20240501 08:12:44.901926 1582 ts_tablet_manager.cc:902] T ac4914417d9e44e7a9591be5386f1d8b P 31811708c6774037b82df44024b229b6: Initiating tablet copy from peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:44.902880 1619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecab521101074a7b80b862642059daf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:44.904896 30672 raft_consensus.cc:1203] T 9820951bffc749139229e2b79c8ab597 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:44.906658 1641 tablet_bootstrap.cc:492] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:44.916878 1623 tablet_copy_client.cc:308] T e6fb30f10c754d3a8e73e124ec59768b P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.131:36305
I20240501 08:12:44.923115 1619 tablet_copy_client.cc:783] T ecab521101074a7b80b862642059daf2 P f1525cbc82a24d13b65f136695c903af: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:44.923388 1619 tablet_copy_client.cc:647] T ecab521101074a7b80b862642059daf2 P f1525cbc82a24d13b65f136695c903af: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:44.925480 28042 tablet_copy_service.cc:140] P aad81047009a4ffcbc450b60030583b5: Received BeginTabletCopySession request for tablet e6fb30f10c754d3a8e73e124ec59768b from peer 9a022e1a501f4d44a0468054d6c3de47 ({username='slave'} at 127.21.84.134:40301)
I20240501 08:12:44.925631 28042 tablet_copy_service.cc:161] P aad81047009a4ffcbc450b60030583b5: Beginning new tablet copy session on tablet e6fb30f10c754d3a8e73e124ec59768b from peer 9a022e1a501f4d44a0468054d6c3de47 at {username='slave'} at 127.21.84.134:40301: session id = 9a022e1a501f4d44a0468054d6c3de47-e6fb30f10c754d3a8e73e124ec59768b
I20240501 08:12:44.928571 28042 tablet_copy_source_session.cc:205] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:44.941243 30672 consensus_queue.cc:237] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:12:44.950822 1623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6fb30f10c754d3a8e73e124ec59768b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:44.966612 1623 tablet_copy_client.cc:783] T e6fb30f10c754d3a8e73e124ec59768b P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:44.971086 1623 tablet_copy_client.cc:647] T e6fb30f10c754d3a8e73e124ec59768b P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:44.972760 1621 ts_tablet_manager.cc:902] T 30130709805e4fa6ae94a46034710863 P 64998323a1f4482d8704bf20594d13a5: Initiating tablet copy from peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:44.980614 30537 raft_consensus.cc:1261] T cb6e886e8cdd451d9a192a771482728e P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:44.981619 362 consensus_queue.cc:1035] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:44.982911 1641 tablet_bootstrap.cc:492] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:44.983479 1641 tablet_bootstrap.cc:492] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5: Bootstrap complete.
I20240501 08:12:44.983788 1641 ts_tablet_manager.cc:1367] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.077s user 0.028s sys 0.028s
I20240501 08:12:44.984839 1641 raft_consensus.cc:348] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
I20240501 08:12:44.985108 1641 raft_consensus.cc:729] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: LEARNER
I20240501 08:12:44.985358 1641 consensus_queue.cc:260] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
I20240501 08:12:44.986724 27891 raft_consensus.cc:1203] T c0e30c62c2d441618195f30f15647487 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:44.992744 28152 raft_consensus.cc:1261] T cb6e886e8cdd451d9a192a771482728e P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:44.996503 1641 ts_tablet_manager.cc:1398] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.013s user 0.003s sys 0.001s
I20240501 08:12:44.997311 28172 tablet_copy_service.cc:342] P 91d11eae5ef6432ca86eb8ed446b6ea6: Request end of tablet copy session 64998323a1f4482d8704bf20594d13a5-98c8e3a6f66a42b08507668f7c3a8af1 received from {username='slave'} at 127.21.84.135:48917
I20240501 08:12:44.997454 28172 tablet_copy_service.cc:433] P 91d11eae5ef6432ca86eb8ed446b6ea6: ending tablet copy session 64998323a1f4482d8704bf20594d13a5-98c8e3a6f66a42b08507668f7c3a8af1 on tablet 98c8e3a6f66a42b08507668f7c3a8af1 with peer 64998323a1f4482d8704bf20594d13a5
W20240501 08:12:44.998927 30607 consensus_peers.cc:485] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 -> Peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Couldn't send request to peer aad81047009a4ffcbc450b60030583b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cb6e886e8cdd451d9a192a771482728e. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:45.001387 362 consensus_queue.cc:1035] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:45.010366 1619 tablet_copy_client.cc:523] T ecab521101074a7b80b862642059daf2 P f1525cbc82a24d13b65f136695c903af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:45.011164 1621 tablet_copy_client.cc:308] T 30130709805e4fa6ae94a46034710863 P 64998323a1f4482d8704bf20594d13a5: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.132:41323
I20240501 08:12:45.012027 28172 tablet_copy_service.cc:140] P 91d11eae5ef6432ca86eb8ed446b6ea6: Received BeginTabletCopySession request for tablet 30130709805e4fa6ae94a46034710863 from peer 64998323a1f4482d8704bf20594d13a5 ({username='slave'} at 127.21.84.135:48917)
I20240501 08:12:45.012184 28172 tablet_copy_service.cc:161] P 91d11eae5ef6432ca86eb8ed446b6ea6: Beginning new tablet copy session on tablet 30130709805e4fa6ae94a46034710863 from peer 64998323a1f4482d8704bf20594d13a5 at {username='slave'} at 127.21.84.135:48917: session id = 64998323a1f4482d8704bf20594d13a5-30130709805e4fa6ae94a46034710863
I20240501 08:12:45.014922 28172 tablet_copy_source_session.cc:205] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:45.015856 1619 tablet_bootstrap.cc:492] T ecab521101074a7b80b862642059daf2 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:45.016273 1621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30130709805e4fa6ae94a46034710863. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:45.024420 1621 tablet_copy_client.cc:783] T 30130709805e4fa6ae94a46034710863 P 64998323a1f4482d8704bf20594d13a5: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:45.024698 1621 tablet_copy_client.cc:647] T 30130709805e4fa6ae94a46034710863 P 64998323a1f4482d8704bf20594d13a5: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:45.026403 1621 tablet_copy_client.cc:523] T 30130709805e4fa6ae94a46034710863 P 64998323a1f4482d8704bf20594d13a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:45.027441 1678 raft_consensus.cc:2932] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:45.029525 30537 raft_consensus.cc:2932] T cb6e886e8cdd451d9a192a771482728e P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:45.030241 1621 tablet_bootstrap.cc:492] T 30130709805e4fa6ae94a46034710863 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:45.038722 27618 catalog_manager.cc:5484] T cb6e886e8cdd451d9a192a771482728e P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New cstate: current_term: 1 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
W20240501 08:12:45.041020 30605 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.132:41323, user_credentials={real_user=slave}} blocked reactor thread for 47701.1us
I20240501 08:12:45.068909 28152 raft_consensus.cc:2932] T cb6e886e8cdd451d9a192a771482728e P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:45.086006 1619 tablet_bootstrap.cc:492] T ecab521101074a7b80b862642059daf2 P f1525cbc82a24d13b65f136695c903af: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:45.086447 1619 tablet_bootstrap.cc:492] T ecab521101074a7b80b862642059daf2 P f1525cbc82a24d13b65f136695c903af: Bootstrap complete.
I20240501 08:12:45.086705 1619 ts_tablet_manager.cc:1367] T ecab521101074a7b80b862642059daf2 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.071s user 0.008s sys 0.002s
I20240501 08:12:45.087499 1619 raft_consensus.cc:348] T ecab521101074a7b80b862642059daf2 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:45.087797 1619 raft_consensus.cc:729] T ecab521101074a7b80b862642059daf2 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: LEARNER
I20240501 08:12:45.088047 1619 consensus_queue.cc:260] T ecab521101074a7b80b862642059daf2 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:45.093315 1619 ts_tablet_manager.cc:1398] T ecab521101074a7b80b862642059daf2 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:12:45.097247 30957 tablet_copy_service.cc:342] P a29cd590999b4f32849ff5689cd53c11: Request end of tablet copy session f1525cbc82a24d13b65f136695c903af-ecab521101074a7b80b862642059daf2 received from {username='slave'} at 127.21.84.130:49339
I20240501 08:12:45.097427 30957 tablet_copy_service.cc:433] P a29cd590999b4f32849ff5689cd53c11: ending tablet copy session f1525cbc82a24d13b65f136695c903af-ecab521101074a7b80b862642059daf2 on tablet ecab521101074a7b80b862642059daf2 with peer f1525cbc82a24d13b65f136695c903af
I20240501 08:12:45.113317 30672 raft_consensus.cc:1203] T 132d1b9866574dcea3beed5835e4a22f P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:45.128505 30800 raft_consensus.cc:1203] T e3d07789edc74dec975e35e9d94cb0ef P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:45.134536 30800 raft_consensus.cc:1203] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240501 08:12:45.187640 30607 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.129:43471, user_credentials={real_user=slave}} blocked reactor thread for 188328us
I20240501 08:12:45.196374 1621 tablet_bootstrap.cc:492] T 30130709805e4fa6ae94a46034710863 P 64998323a1f4482d8704bf20594d13a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:45.196902 1621 tablet_bootstrap.cc:492] T 30130709805e4fa6ae94a46034710863 P 64998323a1f4482d8704bf20594d13a5: Bootstrap complete.
I20240501 08:12:45.197171 1621 ts_tablet_manager.cc:1367] T 30130709805e4fa6ae94a46034710863 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.167s user 0.070s sys 0.074s
I20240501 08:12:45.197995 1621 raft_consensus.cc:348] T 30130709805e4fa6ae94a46034710863 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
I20240501 08:12:45.198835 1621 raft_consensus.cc:729] T 30130709805e4fa6ae94a46034710863 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: LEARNER
I20240501 08:12:45.199081 1621 consensus_queue.cc:260] T 30130709805e4fa6ae94a46034710863 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
I20240501 08:12:45.199733 1621 ts_tablet_manager.cc:1398] T 30130709805e4fa6ae94a46034710863 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:45.200398 28172 tablet_copy_service.cc:342] P 91d11eae5ef6432ca86eb8ed446b6ea6: Request end of tablet copy session 64998323a1f4482d8704bf20594d13a5-30130709805e4fa6ae94a46034710863 received from {username='slave'} at 127.21.84.135:48917
I20240501 08:12:45.200518 28172 tablet_copy_service.cc:433] P 91d11eae5ef6432ca86eb8ed446b6ea6: ending tablet copy session 64998323a1f4482d8704bf20594d13a5-30130709805e4fa6ae94a46034710863 on tablet 30130709805e4fa6ae94a46034710863 with peer 64998323a1f4482d8704bf20594d13a5
I20240501 08:12:45.204934 30537 raft_consensus.cc:1203] T 0e0f87fa9df24b43bedf063a9063bcc5 P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:45.207896 1623 tablet_copy_client.cc:523] T e6fb30f10c754d3a8e73e124ec59768b P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:45.211583 1623 tablet_bootstrap.cc:492] T e6fb30f10c754d3a8e73e124ec59768b P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:45.266955 1419 raft_consensus.cc:1050] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af: attempting to promote NON_VOTER 31811708c6774037b82df44024b229b6 to VOTER
I20240501 08:12:45.267792 1419 consensus_queue.cc:237] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } }
I20240501 08:12:45.273164 27758 consensus_queue.cc:237] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:12:45.277331 1623 tablet_bootstrap.cc:492] T e6fb30f10c754d3a8e73e124ec59768b P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:45.278000 1623 tablet_bootstrap.cc:492] T e6fb30f10c754d3a8e73e124ec59768b P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap complete.
I20240501 08:12:45.278313 1623 ts_tablet_manager.cc:1367] T e6fb30f10c754d3a8e73e124ec59768b P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.067s user 0.036s sys 0.021s
I20240501 08:12:45.279264 1623 raft_consensus.cc:348] T e6fb30f10c754d3a8e73e124ec59768b P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:12:45.279641 1623 raft_consensus.cc:729] T e6fb30f10c754d3a8e73e124ec59768b P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: LEARNER
I20240501 08:12:45.279891 1623 consensus_queue.cc:260] T e6fb30f10c754d3a8e73e124ec59768b P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:12:45.280535 1623 ts_tablet_manager.cc:1398] T e6fb30f10c754d3a8e73e124ec59768b P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:45.283012 28042 tablet_copy_service.cc:342] P aad81047009a4ffcbc450b60030583b5: Request end of tablet copy session 9a022e1a501f4d44a0468054d6c3de47-e6fb30f10c754d3a8e73e124ec59768b received from {username='slave'} at 127.21.84.134:40301
I20240501 08:12:45.283160 28042 tablet_copy_service.cc:433] P aad81047009a4ffcbc450b60030583b5: ending tablet copy session 9a022e1a501f4d44a0468054d6c3de47-e6fb30f10c754d3a8e73e124ec59768b on tablet e6fb30f10c754d3a8e73e124ec59768b with peer 9a022e1a501f4d44a0468054d6c3de47
I20240501 08:12:45.289217 28152 raft_consensus.cc:1261] T f2baff8656804ce2bfb64a9280e16855 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:45.290161 1089 consensus_queue.cc:1035] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:45.291318 30669 raft_consensus.cc:1203] T e6fb30f10c754d3a8e73e124ec59768b P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:45.292552 28020 raft_consensus.cc:1261] T f2baff8656804ce2bfb64a9280e16855 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:45.295348 1426 consensus_queue.cc:1035] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:12:45.298630 27694 consensus_peers.cc:485] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 -> Peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Couldn't send request to peer 9a022e1a501f4d44a0468054d6c3de47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f2baff8656804ce2bfb64a9280e16855. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:45.302325 1426 raft_consensus.cc:2932] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:45.307227 28020 raft_consensus.cc:2932] T f2baff8656804ce2bfb64a9280e16855 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:45.310464 28149 raft_consensus.cc:2932] T f2baff8656804ce2bfb64a9280e16855 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:45.312587 1619 ts_tablet_manager.cc:902] T b605254185d140d8ab90fcaa5229f168 P f1525cbc82a24d13b65f136695c903af: Initiating tablet copy from peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:45.317773 1619 tablet_copy_client.cc:308] T b605254185d140d8ab90fcaa5229f168 P f1525cbc82a24d13b65f136695c903af: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.136:46845
I20240501 08:12:45.322917 30957 tablet_copy_service.cc:140] P a29cd590999b4f32849ff5689cd53c11: Received BeginTabletCopySession request for tablet b605254185d140d8ab90fcaa5229f168 from peer f1525cbc82a24d13b65f136695c903af ({username='slave'} at 127.21.84.130:49339)
I20240501 08:12:45.323225 30957 tablet_copy_service.cc:161] P a29cd590999b4f32849ff5689cd53c11: Beginning new tablet copy session on tablet b605254185d140d8ab90fcaa5229f168 from peer f1525cbc82a24d13b65f136695c903af at {username='slave'} at 127.21.84.130:49339: session id = f1525cbc82a24d13b65f136695c903af-b605254185d140d8ab90fcaa5229f168
I20240501 08:12:45.330116 30957 tablet_copy_source_session.cc:205] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:45.334993 27619 catalog_manager.cc:5484] T f2baff8656804ce2bfb64a9280e16855 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: config changed from index -1 to 4, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:45.393606 27758 consensus_queue.cc:237] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
W20240501 08:12:45.404605 27694 consensus_peers.cc:485] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 -> Peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Couldn't send request to peer 91d11eae5ef6432ca86eb8ed446b6ea6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3aa09c722530431384e5e7ee9759230c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:45.406092 30537 raft_consensus.cc:1261] T 3aa09c722530431384e5e7ee9759230c P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:45.406375 30800 raft_consensus.cc:1261] T 3aa09c722530431384e5e7ee9759230c P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:45.407298 1426 consensus_queue.cc:1035] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:45.407886 1426 consensus_queue.cc:1035] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:12:45.412554 27825 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.134:39933, user_credentials={real_user=slave}} blocked reactor thread for 119552us
I20240501 08:12:45.429194 1619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b605254185d140d8ab90fcaa5229f168. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:45.437201 1619 tablet_copy_client.cc:783] T b605254185d140d8ab90fcaa5229f168 P f1525cbc82a24d13b65f136695c903af: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:45.436642 1089 raft_consensus.cc:2932] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:12:45.437474 1619 tablet_copy_client.cc:647] T b605254185d140d8ab90fcaa5229f168 P f1525cbc82a24d13b65f136695c903af: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:45.439105 30536 raft_consensus.cc:2932] T 3aa09c722530431384e5e7ee9759230c P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:12:45.442404 1619 tablet_copy_client.cc:523] T b605254185d140d8ab90fcaa5229f168 P f1525cbc82a24d13b65f136695c903af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:45.447940 1619 tablet_bootstrap.cc:492] T b605254185d140d8ab90fcaa5229f168 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:45.455788 1430 raft_consensus.cc:1050] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af: attempting to promote NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 to VOTER
I20240501 08:12:45.456996 1430 consensus_queue.cc:237] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:12:45.460325 30803 raft_consensus.cc:2932] T 3aa09c722530431384e5e7ee9759230c P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:12:45.464859 27618 catalog_manager.cc:5484] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:45.469062 28020 raft_consensus.cc:1261] T 7f7211c8c0e041bf89930aa50a4438ea P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:45.469826 1704 consensus_queue.cc:1035] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:45.477861 28018 raft_consensus.cc:1261] T 9820951bffc749139229e2b79c8ab597 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:45.478171 27758 raft_consensus.cc:1261] T 7f7211c8c0e041bf89930aa50a4438ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:45.479396 27758 raft_consensus.cc:1261] T 9820951bffc749139229e2b79c8ab597 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:45.480310 30537 raft_consensus.cc:1261] T 7f7211c8c0e041bf89930aa50a4438ea P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:45.480343 1698 consensus_queue.cc:1035] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:45.480873 1716 consensus_queue.cc:1035] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:45.481452 1711 consensus_queue.cc:1035] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:45.488664 27891 raft_consensus.cc:1203] T ecab521101074a7b80b862642059daf2 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:45.493628 1698 consensus_queue.cc:1035] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:45.498756 30669 raft_consensus.cc:1261] T 9820951bffc749139229e2b79c8ab597 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:45.513731 1704 consensus_queue.cc:1035] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:45.527194 30803 raft_consensus.cc:1203] T 30130709805e4fa6ae94a46034710863 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:45.565897 1419 raft_consensus.cc:2932] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:45.567552 1543 raft_consensus.cc:1050] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11: attempting to promote NON_VOTER f1525cbc82a24d13b65f136695c903af to VOTER
I20240501 08:12:45.567272 30669 raft_consensus.cc:2932] T 9820951bffc749139229e2b79c8ab597 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:45.568552 1543 consensus_queue.cc:237] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:45.572525 30671 raft_consensus.cc:1261] T c0e30c62c2d441618195f30f15647487 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:45.573729 1379 consensus_queue.cc:1035] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:45.575667 27616 catalog_manager.cc:5484] T 9820951bffc749139229e2b79c8ab597 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 4 to 5, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:45.577667 840 raft_consensus.cc:1050] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6: attempting to promote NON_VOTER 64998323a1f4482d8704bf20594d13a5 to VOTER
I20240501 08:12:45.581400 840 consensus_queue.cc:237] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:12:45.584200 27759 raft_consensus.cc:1261] T c0e30c62c2d441618195f30f15647487 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:45.585469 1379 consensus_queue.cc:1035] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 08:12:45.594681 27891 raft_consensus.cc:1261] T c0e30c62c2d441618195f30f15647487 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:45.595436 1543 consensus_queue.cc:1035] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:45.596309 1712 raft_consensus.cc:2932] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:45.601472 27760 raft_consensus.cc:2932] T 9820951bffc749139229e2b79c8ab597 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:45.614500 27759 raft_consensus.cc:2932] T 7f7211c8c0e041bf89930aa50a4438ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:45.624542 28018 raft_consensus.cc:2932] T 7f7211c8c0e041bf89930aa50a4438ea P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:45.635330 1089 raft_consensus.cc:1050] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43: attempting to promote NON_VOTER 64998323a1f4482d8704bf20594d13a5 to VOTER
I20240501 08:12:45.636229 1089 consensus_queue.cc:237] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:12:45.633730 27616 catalog_manager.cc:5484] T 7f7211c8c0e041bf89930aa50a4438ea P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 4 to 5, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:45.650396 27890 raft_consensus.cc:1261] T e3d07789edc74dec975e35e9d94cb0ef P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:45.653066 1089 consensus_queue.cc:1035] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:45.663018 1740 raft_consensus.cc:1050] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6: attempting to promote NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 to VOTER
I20240501 08:12:45.664073 1740 consensus_queue.cc:237] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:12:45.668454 28152 raft_consensus.cc:1261] T e3d07789edc74dec975e35e9d94cb0ef P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:45.669349 1089 consensus_queue.cc:1035] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:45.674721 27890 raft_consensus.cc:1261] T 98c8e3a6f66a42b08507668f7c3a8af1 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:45.675727 1741 consensus_queue.cc:1035] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:45.677258 30803 raft_consensus.cc:1261] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:45.687753 1199 consensus_queue.cc:1035] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 08:12:45.702752 27827 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.131:36305, user_credentials={real_user=slave}} blocked reactor thread for 78116.8us
I20240501 08:12:45.705837 1582 tablet_copy_client.cc:308] T ac4914417d9e44e7a9591be5386f1d8b P 31811708c6774037b82df44024b229b6: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.129:43471
I20240501 08:12:45.706651 27781 tablet_copy_service.cc:140] P 6331f77dae9d452184e8cd0cbf5f1b43: Received BeginTabletCopySession request for tablet ac4914417d9e44e7a9591be5386f1d8b from peer 31811708c6774037b82df44024b229b6 ({username='slave'} at 127.21.84.133:35347)
I20240501 08:12:45.706817 27781 tablet_copy_service.cc:161] P 6331f77dae9d452184e8cd0cbf5f1b43: Beginning new tablet copy session on tablet ac4914417d9e44e7a9591be5386f1d8b from peer 31811708c6774037b82df44024b229b6 at {username='slave'} at 127.21.84.133:35347: session id = 31811708c6774037b82df44024b229b6-ac4914417d9e44e7a9591be5386f1d8b
I20240501 08:12:45.708467 30537 raft_consensus.cc:2932] T 7f7211c8c0e041bf89930aa50a4438ea P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:45.709684 27781 tablet_copy_source_session.cc:205] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:45.716766 1582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac4914417d9e44e7a9591be5386f1d8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:45.717669 30669 raft_consensus.cc:1261] T 132d1b9866574dcea3beed5835e4a22f P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:45.718266 1735 consensus_queue.cc:1035] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:45.728190 1734 ts_tablet_manager.cc:902] T cb6e886e8cdd451d9a192a771482728e P aad81047009a4ffcbc450b60030583b5: Initiating tablet copy from peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:45.730842 1619 tablet_bootstrap.cc:492] T b605254185d140d8ab90fcaa5229f168 P f1525cbc82a24d13b65f136695c903af: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:45.731284 1619 tablet_bootstrap.cc:492] T b605254185d140d8ab90fcaa5229f168 P f1525cbc82a24d13b65f136695c903af: Bootstrap complete.
I20240501 08:12:45.731571 1619 ts_tablet_manager.cc:1367] T b605254185d140d8ab90fcaa5229f168 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.284s user 0.002s sys 0.006s
I20240501 08:12:45.736377 30934 raft_consensus.cc:1261] T 98c8e3a6f66a42b08507668f7c3a8af1 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:45.737454 28020 raft_consensus.cc:1261] T 132d1b9866574dcea3beed5835e4a22f P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:45.737883 1378 raft_consensus.cc:2932] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:45.742632 27760 raft_consensus.cc:2932] T c0e30c62c2d441618195f30f15647487 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:45.744998 1741 consensus_queue.cc:1035] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:45.745684 840 consensus_queue.cc:1035] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:45.748016 27761 raft_consensus.cc:1261] T 132d1b9866574dcea3beed5835e4a22f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:45.749621 27890 raft_consensus.cc:2932] T c0e30c62c2d441618195f30f15647487 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:45.750615 27891 consensus_queue.cc:237] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } }
I20240501 08:12:45.752926 30537 raft_consensus.cc:1261] T 7f7211c8c0e041bf89930aa50a4438ea P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:45.753472 1704 consensus_queue.cc:1035] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:45.755833 1704 raft_consensus.cc:2932] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:45.758575 30537 raft_consensus.cc:2932] T 7f7211c8c0e041bf89930aa50a4438ea P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:45.764184 27612 catalog_manager.cc:5484] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:45.765368 1555 raft_consensus.cc:1050] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5: attempting to promote NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 to VOTER
I20240501 08:12:45.766999 27607 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7f7211c8c0e041bf89930aa50a4438ea with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:45.768386 1555 consensus_queue.cc:237] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:12:45.774689 27611 catalog_manager.cc:5484] T 7f7211c8c0e041bf89930aa50a4438ea P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:45.778308 1379 raft_consensus.cc:1050] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11: attempting to promote NON_VOTER f1525cbc82a24d13b65f136695c903af to VOTER
I20240501 08:12:45.779167 1379 consensus_queue.cc:237] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:45.782367 28000 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f7211c8c0e041bf89930aa50a4438ea with delete_type TABLET_DATA_TOMBSTONED (TS aad81047009a4ffcbc450b60030583b5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:45.787678 1757 tablet_replica.cc:330] T 7f7211c8c0e041bf89930aa50a4438ea P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:45.788987 1757 raft_consensus.cc:2229] T 7f7211c8c0e041bf89930aa50a4438ea P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:45.790197 1757 raft_consensus.cc:2258] T 7f7211c8c0e041bf89930aa50a4438ea P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:45.791471 1757 ts_tablet_manager.cc:1875] T 7f7211c8c0e041bf89930aa50a4438ea P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:45.797878 30800 raft_consensus.cc:1261] T e3d07789edc74dec975e35e9d94cb0ef P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:45.800283 1707 raft_consensus.cc:2932] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:45.805269 27758 raft_consensus.cc:1261] T 7f7211c8c0e041bf89930aa50a4438ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:45.806030 1711 consensus_queue.cc:1035] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:45.810374 27890 raft_consensus.cc:2932] T e3d07789edc74dec975e35e9d94cb0ef P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:45.813191 1757 ts_tablet_manager.cc:1888] T 7f7211c8c0e041bf89930aa50a4438ea P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:45.813593 1757 log.cc:1196] T 7f7211c8c0e041bf89930aa50a4438ea P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/7f7211c8c0e041bf89930aa50a4438ea
I20240501 08:12:45.814586 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet 7f7211c8c0e041bf89930aa50a4438ea (table rebalance_test_table_1 [id=0ee45bbb25734f82b00abd2da25300d1]) successfully deleted
I20240501 08:12:45.814760 28151 raft_consensus.cc:2932] T e3d07789edc74dec975e35e9d94cb0ef P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:45.814788 1747 raft_consensus.cc:2932] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:45.816268 27611 catalog_manager.cc:5484] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:45.826400 30802 raft_consensus.cc:2932] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:45.826934 27890 raft_consensus.cc:2932] T 98c8e3a6f66a42b08507668f7c3a8af1 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:45.838593 27612 catalog_manager.cc:5484] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index 2 to 3, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:45.842242 27759 raft_consensus.cc:2932] T 7f7211c8c0e041bf89930aa50a4438ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:45.860777 1426 raft_consensus.cc:1050] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43: attempting to promote NON_VOTER 31811708c6774037b82df44024b229b6 to VOTER
I20240501 08:12:45.861608 1426 consensus_queue.cc:237] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } }
I20240501 08:12:45.865473 30537 raft_consensus.cc:1261] T 0e0f87fa9df24b43bedf063a9063bcc5 P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:45.865859 27890 raft_consensus.cc:1261] T 0e0f87fa9df24b43bedf063a9063bcc5 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:45.866034 1089 consensus_queue.cc:1035] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:45.866812 28018 raft_consensus.cc:1261] T 0e0f87fa9df24b43bedf063a9063bcc5 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:45.870064 1758 ts_tablet_manager.cc:902] T 3aa09c722530431384e5e7ee9759230c P 91d11eae5ef6432ca86eb8ed446b6ea6: Initiating tablet copy from peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:45.873453 27760 raft_consensus.cc:1261] T ecab521101074a7b80b862642059daf2 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:45.873687 30537 raft_consensus.cc:1261] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:45.874325 1543 consensus_queue.cc:1035] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:45.874662 1089 consensus_queue.cc:1035] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:45.874958 1543 consensus_queue.cc:1035] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:45.875186 1089 consensus_queue.cc:1035] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:45.880650 28022 raft_consensus.cc:2932] T 9820951bffc749139229e2b79c8ab597 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:45.881428 27890 raft_consensus.cc:1261] T ecab521101074a7b80b862642059daf2 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:45.882043 1730 consensus_queue.cc:1035] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:45.892757 1619 raft_consensus.cc:348] T b605254185d140d8ab90fcaa5229f168 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:45.893129 1619 raft_consensus.cc:729] T b605254185d140d8ab90fcaa5229f168 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: LEARNER
I20240501 08:12:45.893371 1619 consensus_queue.cc:260] T b605254185d140d8ab90fcaa5229f168 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:45.913408 1738 consensus_queue.cc:1035] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:45.918207 1089 consensus_queue.cc:1035] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:45.923018 30672 raft_consensus.cc:2932] T c0e30c62c2d441618195f30f15647487 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:45.935513 1619 ts_tablet_manager.cc:1398] T b605254185d140d8ab90fcaa5229f168 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.204s user 0.002s sys 0.002s
I20240501 08:12:45.936286 30957 tablet_copy_service.cc:342] P a29cd590999b4f32849ff5689cd53c11: Request end of tablet copy session f1525cbc82a24d13b65f136695c903af-b605254185d140d8ab90fcaa5229f168 received from {username='slave'} at 127.21.84.130:49339
I20240501 08:12:45.936429 30957 tablet_copy_service.cc:433] P a29cd590999b4f32849ff5689cd53c11: ending tablet copy session f1525cbc82a24d13b65f136695c903af-b605254185d140d8ab90fcaa5229f168 on tablet b605254185d140d8ab90fcaa5229f168 with peer f1525cbc82a24d13b65f136695c903af
I20240501 08:12:45.936067 1543 raft_consensus.cc:2932] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:45.939157 30672 raft_consensus.cc:1261] T e6fb30f10c754d3a8e73e124ec59768b P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:45.939867 877 consensus_queue.cc:1035] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:45.947142 27892 consensus_queue.cc:237] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:12:45.950371 27615 catalog_manager.cc:5484] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:45.951134 30672 raft_consensus.cc:1261] T 9820951bffc749139229e2b79c8ab597 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:45.951961 27891 raft_consensus.cc:2932] T ecab521101074a7b80b862642059daf2 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:45.956697 1711 consensus_queue.cc:1035] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:45.957037 30536 raft_consensus.cc:2932] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:45.959054 28151 raft_consensus.cc:1261] T e6fb30f10c754d3a8e73e124ec59768b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:45.960078 30935 raft_consensus.cc:2932] T 98c8e3a6f66a42b08507668f7c3a8af1 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:45.967485 1711 raft_consensus.cc:2932] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:45.970741 30670 raft_consensus.cc:2932] T 9820951bffc749139229e2b79c8ab597 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:45.971398 1747 raft_consensus.cc:2932] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:45.977905 27760 raft_consensus.cc:1261] T e6fb30f10c754d3a8e73e124ec59768b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:45.977859 27607 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9820951bffc749139229e2b79c8ab597 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:45.978621 30668 raft_consensus.cc:2932] T 132d1b9866574dcea3beed5835e4a22f P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:45.983994 27615 catalog_manager.cc:5484] T 9820951bffc749139229e2b79c8ab597 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 5 to 6, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:45.991941 1555 consensus_queue.cc:1035] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:45.995057 30800 raft_consensus.cc:2932] T e3d07789edc74dec975e35e9d94cb0ef P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:46.000408 28016 raft_consensus.cc:1261] T 9820951bffc749139229e2b79c8ab597 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:46.001302 1711 consensus_queue.cc:1035] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:46.002848 28022 raft_consensus.cc:2932] T 9820951bffc749139229e2b79c8ab597 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:46.005887 1555 consensus_queue.cc:1035] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:46.008435 28016 raft_consensus.cc:2932] T 132d1b9866574dcea3beed5835e4a22f P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:46.002750 27616 catalog_manager.cc:5484] T 132d1b9866574dcea3beed5835e4a22f P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 4 to 5, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:46.028191 1582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.21.84.129:35413 (request call id 403) took 1137 ms. Trace:
I20240501 08:12:46.034619 1555 raft_consensus.cc:2932] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:46.035317 27759 raft_consensus.cc:2932] T e6fb30f10c754d3a8e73e124ec59768b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:46.038784 1734 tablet_copy_client.cc:308] T cb6e886e8cdd451d9a192a771482728e P aad81047009a4ffcbc450b60030583b5: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.134:39933
I20240501 08:12:46.028403 1582 rpcz_store.cc:276] 0501 08:12:44.890319 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 08:12:44.890387 (+ 68us) service_pool.cc:225] Handling call
0501 08:12:44.902023 (+ 11636us) ts_tablet_manager.cc:903] T ac4914417d9e44e7a9591be5386f1d8b P 31811708c6774037b82df44024b229b6: Initiating tablet copy from peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
0501 08:12:45.714019 (+811996us) tablet_copy_client.cc:353] Tablet copy session begun
0501 08:12:45.721102 (+ 7083us) tablet_metadata.cc:641] Metadata flushed
0501 08:12:45.721215 (+ 113us) tablet_copy_client.cc:482] Wrote new tablet metadata
0501 08:12:46.027860 (+306645us) tablet_copy_client.cc:403] Wrote new consensus metadata
0501 08:12:46.027995 (+ 135us) tablet_replica.cc:205] Creating consensus instance
0501 08:12:46.028096 (+ 101us) ts_tablet_manager.cc:958] Replica registered.
0501 08:12:46.028124 (+ 28us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":101,"tablet-copy.queue_time_us":11326,"thread_start_us":848665,"threads_started":5}
I20240501 08:12:46.046842 1582 tablet_copy_client.cc:783] T ac4914417d9e44e7a9591be5386f1d8b P 31811708c6774037b82df44024b229b6: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:46.047134 1582 tablet_copy_client.cc:647] T ac4914417d9e44e7a9591be5386f1d8b P 31811708c6774037b82df44024b229b6: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:46.039541 30693 tablet_copy_service.cc:140] P 9a022e1a501f4d44a0468054d6c3de47: Received BeginTabletCopySession request for tablet cb6e886e8cdd451d9a192a771482728e from peer aad81047009a4ffcbc450b60030583b5 ({username='slave'} at 127.21.84.131:44127)
I20240501 08:12:46.047865 30693 tablet_copy_service.cc:161] P 9a022e1a501f4d44a0468054d6c3de47: Beginning new tablet copy session on tablet cb6e886e8cdd451d9a192a771482728e from peer aad81047009a4ffcbc450b60030583b5 at {username='slave'} at 127.21.84.131:44127: session id = aad81047009a4ffcbc450b60030583b5-cb6e886e8cdd451d9a192a771482728e
I20240501 08:12:46.048164 27615 catalog_manager.cc:5484] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 4 to 5, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:46.036836 1768 ts_tablet_manager.cc:902] T f2baff8656804ce2bfb64a9280e16855 P 9a022e1a501f4d44a0468054d6c3de47: Initiating tablet copy from peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:46.053328 28149 raft_consensus.cc:2932] T e6fb30f10c754d3a8e73e124ec59768b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:46.055840 30693 tablet_copy_source_session.cc:205] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:46.040055 30669 raft_consensus.cc:2932] T e6fb30f10c754d3a8e73e124ec59768b P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:46.064302 1734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb6e886e8cdd451d9a192a771482728e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:46.068349 1758 tablet_copy_client.cc:308] T 3aa09c722530431384e5e7ee9759230c P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.129:43471
I20240501 08:12:46.071776 28149 consensus_queue.cc:237] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:12:46.071748 28022 consensus_queue.cc:237] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:12:46.076201 1735 raft_consensus.cc:1050] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6: attempting to promote NON_VOTER 64998323a1f4482d8704bf20594d13a5 to VOTER
I20240501 08:12:46.077023 1735 consensus_queue.cc:237] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:12:46.078761 28151 raft_consensus.cc:1261] T e6fb30f10c754d3a8e73e124ec59768b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:46.084254 27741 tablet_service.cc:1514] Processing DeleteTablet for tablet 9820951bffc749139229e2b79c8ab597 with delete_type TABLET_DATA_TOMBSTONED (TS 6331f77dae9d452184e8cd0cbf5f1b43 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:46.084991 1763 raft_consensus.cc:2932] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:46.094130 1555 consensus_queue.cc:1035] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:46.097849 27781 tablet_copy_service.cc:140] P 6331f77dae9d452184e8cd0cbf5f1b43: Received BeginTabletCopySession request for tablet 3aa09c722530431384e5e7ee9759230c from peer 91d11eae5ef6432ca86eb8ed446b6ea6 ({username='slave'} at 127.21.84.132:59727)
I20240501 08:12:46.098043 27781 tablet_copy_service.cc:161] P 6331f77dae9d452184e8cd0cbf5f1b43: Beginning new tablet copy session on tablet 3aa09c722530431384e5e7ee9759230c from peer 91d11eae5ef6432ca86eb8ed446b6ea6 at {username='slave'} at 127.21.84.132:59727: session id = 91d11eae5ef6432ca86eb8ed446b6ea6-3aa09c722530431384e5e7ee9759230c
I20240501 08:12:46.101203 27781 tablet_copy_source_session.cc:205] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:46.107151 1772 raft_consensus.cc:2932] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:46.111266 30672 raft_consensus.cc:1261] T e6fb30f10c754d3a8e73e124ec59768b P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:46.112319 877 consensus_queue.cc:1035] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:46.113360 30536 raft_consensus.cc:2932] T 0e0f87fa9df24b43bedf063a9063bcc5 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:46.115234 30800 raft_consensus.cc:1261] T 30130709805e4fa6ae94a46034710863 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:46.117118 1582 tablet_copy_client.cc:523] T ac4914417d9e44e7a9591be5386f1d8b P 31811708c6774037b82df44024b229b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:46.118182 1735 consensus_queue.cc:1035] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:46.119079 30669 raft_consensus.cc:2932] T e6fb30f10c754d3a8e73e124ec59768b P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:46.122901 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e6fb30f10c754d3a8e73e124ec59768b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:46.126317 1734 tablet_copy_client.cc:783] T cb6e886e8cdd451d9a192a771482728e P aad81047009a4ffcbc450b60030583b5: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:46.126688 1734 tablet_copy_client.cc:647] T cb6e886e8cdd451d9a192a771482728e P aad81047009a4ffcbc450b60030583b5: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:46.127558 27616 catalog_manager.cc:5484] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 5 to 6, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:46.128553 30672 raft_consensus.cc:1261] T 132d1b9866574dcea3beed5835e4a22f P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:46.131855 1738 consensus_queue.cc:1035] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:46.135273 28016 raft_consensus.cc:1261] T 30130709805e4fa6ae94a46034710863 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:46.135505 27618 catalog_manager.cc:5484] T 0e0f87fa9df24b43bedf063a9063bcc5 P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index 4 to 5, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:46.138374 1738 raft_consensus.cc:2932] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:46.142971 30672 raft_consensus.cc:2932] T 132d1b9866574dcea3beed5835e4a22f P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:46.147325 28018 raft_consensus.cc:1261] T 132d1b9866574dcea3beed5835e4a22f P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:46.147935 28016 raft_consensus.cc:2932] T 0e0f87fa9df24b43bedf063a9063bcc5 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:46.148350 1582 tablet_bootstrap.cc:492] T ac4914417d9e44e7a9591be5386f1d8b P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:46.150820 1791 tablet_replica.cc:330] T 9820951bffc749139229e2b79c8ab597 P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:46.151333 1791 raft_consensus.cc:2229] T 9820951bffc749139229e2b79c8ab597 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:46.151793 27612 catalog_manager.cc:5484] T 132d1b9866574dcea3beed5835e4a22f P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 5 to 6, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:46.159902 1734 tablet_copy_client.cc:523] T cb6e886e8cdd451d9a192a771482728e P aad81047009a4ffcbc450b60030583b5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:46.163475 27604 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 132d1b9866574dcea3beed5835e4a22f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:46.166270 1791 raft_consensus.cc:2258] T 9820951bffc749139229e2b79c8ab597 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:46.170738 1791 ts_tablet_manager.cc:1875] T 9820951bffc749139229e2b79c8ab597 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:46.172102 1734 tablet_bootstrap.cc:492] T cb6e886e8cdd451d9a192a771482728e P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:46.175446 27891 raft_consensus.cc:2932] T 0e0f87fa9df24b43bedf063a9063bcc5 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:46.175412 27759 raft_consensus.cc:1261] T 30130709805e4fa6ae94a46034710863 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:46.178373 28149 raft_consensus.cc:2932] T e6fb30f10c754d3a8e73e124ec59768b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:46.178732 27741 tablet_service.cc:1514] Processing DeleteTablet for tablet e6fb30f10c754d3a8e73e124ec59768b with delete_type TABLET_DATA_TOMBSTONED (TS 6331f77dae9d452184e8cd0cbf5f1b43 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:46.179941 1758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aa09c722530431384e5e7ee9759230c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:46.182415 27741 tablet_service.cc:1514] Processing DeleteTablet for tablet 132d1b9866574dcea3beed5835e4a22f with delete_type TABLET_DATA_TOMBSTONED (TS 6331f77dae9d452184e8cd0cbf5f1b43 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:46.185806 1738 consensus_queue.cc:1035] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:46.186414 1758 tablet_copy_client.cc:783] T 3aa09c722530431384e5e7ee9759230c P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:46.186758 1758 tablet_copy_client.cc:647] T 3aa09c722530431384e5e7ee9759230c P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:46.186918 1738 consensus_queue.cc:1035] T 132d1b9866574dcea3beed5835e4a22f P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:46.187167 27759 consensus_queue.cc:237] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } }
I20240501 08:12:46.189283 27760 raft_consensus.cc:2932] T ecab521101074a7b80b862642059daf2 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:46.191474 1768 tablet_copy_client.cc:308] T f2baff8656804ce2bfb64a9280e16855 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.129:43471
I20240501 08:12:46.192160 27781 tablet_copy_service.cc:140] P 6331f77dae9d452184e8cd0cbf5f1b43: Received BeginTabletCopySession request for tablet f2baff8656804ce2bfb64a9280e16855 from peer 9a022e1a501f4d44a0468054d6c3de47 ({username='slave'} at 127.21.84.134:48929)
I20240501 08:12:46.192292 27781 tablet_copy_service.cc:161] P 6331f77dae9d452184e8cd0cbf5f1b43: Beginning new tablet copy session on tablet f2baff8656804ce2bfb64a9280e16855 from peer 9a022e1a501f4d44a0468054d6c3de47 at {username='slave'} at 127.21.84.134:48929: session id = 9a022e1a501f4d44a0468054d6c3de47-f2baff8656804ce2bfb64a9280e16855
I20240501 08:12:46.193511 30536 raft_consensus.cc:1261] T 0e0f87fa9df24b43bedf063a9063bcc5 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:46.195286 27781 tablet_copy_source_session.cc:205] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:46.197358 1768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2baff8656804ce2bfb64a9280e16855. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:46.207964 1766 consensus_queue.cc:1035] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:46.212014 1089 raft_consensus.cc:2932] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:46.218022 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0e0f87fa9df24b43bedf063a9063bcc5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:46.217190 30536 raft_consensus.cc:2932] T 0e0f87fa9df24b43bedf063a9063bcc5 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:46.223124 27890 raft_consensus.cc:1261] T 0e0f87fa9df24b43bedf063a9063bcc5 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:46.224171 1089 consensus_queue.cc:1035] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:46.226627 27890 raft_consensus.cc:2932] T 0e0f87fa9df24b43bedf063a9063bcc5 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:46.231115 1791 ts_tablet_manager.cc:1888] T 9820951bffc749139229e2b79c8ab597 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:46.231372 27615 catalog_manager.cc:5484] T 0e0f87fa9df24b43bedf063a9063bcc5 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:46.237910 28000 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e0f87fa9df24b43bedf063a9063bcc5 with delete_type TABLET_DATA_TOMBSTONED (TS aad81047009a4ffcbc450b60030583b5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:46.238202 1768 tablet_copy_client.cc:783] T f2baff8656804ce2bfb64a9280e16855 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:46.238288 1757 tablet_replica.cc:330] T 0e0f87fa9df24b43bedf063a9063bcc5 P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:46.238507 1768 tablet_copy_client.cc:647] T f2baff8656804ce2bfb64a9280e16855 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:46.238492 1757 raft_consensus.cc:2229] T 0e0f87fa9df24b43bedf063a9063bcc5 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:46.238651 1791 log.cc:1196] T 9820951bffc749139229e2b79c8ab597 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/9820951bffc749139229e2b79c8ab597
I20240501 08:12:46.238703 1757 raft_consensus.cc:2258] T 0e0f87fa9df24b43bedf063a9063bcc5 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:46.239449 1791 tablet_replica.cc:330] T e6fb30f10c754d3a8e73e124ec59768b P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:46.239461 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet 9820951bffc749139229e2b79c8ab597 (table rebalance_test_table_13 [id=86d1b715252040a3bc0f714b3db3f4ff]) successfully deleted
I20240501 08:12:46.239620 1791 raft_consensus.cc:2229] T e6fb30f10c754d3a8e73e124ec59768b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:46.239801 1791 raft_consensus.cc:2258] T e6fb30f10c754d3a8e73e124ec59768b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:46.240391 1757 ts_tablet_manager.cc:1875] T 0e0f87fa9df24b43bedf063a9063bcc5 P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:46.240897 28018 raft_consensus.cc:2932] T 132d1b9866574dcea3beed5835e4a22f P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:46.241575 1768 tablet_copy_client.cc:523] T f2baff8656804ce2bfb64a9280e16855 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:46.248907 27890 raft_consensus.cc:1203] T b605254185d140d8ab90fcaa5229f168 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:46.250613 1791 ts_tablet_manager.cc:1875] T e6fb30f10c754d3a8e73e124ec59768b P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:46.257663 1757 ts_tablet_manager.cc:1888] T 0e0f87fa9df24b43bedf063a9063bcc5 P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:46.257920 1757 log.cc:1196] T 0e0f87fa9df24b43bedf063a9063bcc5 P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/0e0f87fa9df24b43bedf063a9063bcc5
I20240501 08:12:46.258733 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet 0e0f87fa9df24b43bedf063a9063bcc5 (table rebalance_test_table_11 [id=05c446d8d19b4077b3441b0349357c6e]) successfully deleted
I20240501 08:12:46.260124 1738 consensus_queue.cc:1035] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:46.261080 1758 tablet_copy_client.cc:523] T 3aa09c722530431384e5e7ee9759230c P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:46.264540 1758 tablet_bootstrap.cc:492] T 3aa09c722530431384e5e7ee9759230c P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:46.269678 1735 raft_consensus.cc:2932] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:46.271339 1768 tablet_bootstrap.cc:492] T f2baff8656804ce2bfb64a9280e16855 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:46.275446 30800 raft_consensus.cc:2932] T 30130709805e4fa6ae94a46034710863 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:46.289227 1791 ts_tablet_manager.cc:1888] T e6fb30f10c754d3a8e73e124ec59768b P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:46.289644 1791 log.cc:1196] T e6fb30f10c754d3a8e73e124ec59768b P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/e6fb30f10c754d3a8e73e124ec59768b
I20240501 08:12:46.289844 28016 raft_consensus.cc:2932] T 30130709805e4fa6ae94a46034710863 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:46.291730 1758 tablet_bootstrap.cc:492] T 3aa09c722530431384e5e7ee9759230c P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:46.292097 1758 tablet_bootstrap.cc:492] T 3aa09c722530431384e5e7ee9759230c P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap complete.
I20240501 08:12:46.292354 1758 ts_tablet_manager.cc:1367] T 3aa09c722530431384e5e7ee9759230c P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.006s
I20240501 08:12:46.293251 1758 raft_consensus.cc:348] T 3aa09c722530431384e5e7ee9759230c P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:12:46.294433 1791 tablet_replica.cc:330] T 132d1b9866574dcea3beed5835e4a22f P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:46.294565 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet e6fb30f10c754d3a8e73e124ec59768b (table rebalance_test_table_19 [id=4fc7c32c4ee342e5804c1a70b815d952]) successfully deleted
I20240501 08:12:46.297169 1791 raft_consensus.cc:2229] T 132d1b9866574dcea3beed5835e4a22f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:46.297606 1791 pending_rounds.cc:62] T 132d1b9866574dcea3beed5835e4a22f P 6331f77dae9d452184e8cd0cbf5f1b43: Trying to abort 1 pending ops.
I20240501 08:12:46.297318 27618 catalog_manager.cc:5484] T 30130709805e4fa6ae94a46034710863 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:46.297972 1791 pending_rounds.cc:68] T 132d1b9866574dcea3beed5835e4a22f P 6331f77dae9d452184e8cd0cbf5f1b43: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7022801574557388800 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "132d1b9866574dcea3beed5835e4a22f" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } } }
I20240501 08:12:46.293495 1758 raft_consensus.cc:729] T 3aa09c722530431384e5e7ee9759230c P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: LEARNER
I20240501 08:12:46.300282 1791 raft_consensus.cc:2894] T 132d1b9866574dcea3beed5835e4a22f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240501 08:12:46.300122 1758 consensus_queue.cc:260] T 3aa09c722530431384e5e7ee9759230c P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:12:46.300851 1758 ts_tablet_manager.cc:1398] T 3aa09c722530431384e5e7ee9759230c P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.008s user 0.003s sys 0.001s
I20240501 08:12:46.302290 1791 raft_consensus.cc:2866] T 132d1b9866574dcea3beed5835e4a22f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 08:12:46.306071 1791 raft_consensus.cc:2258] T 132d1b9866574dcea3beed5835e4a22f P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:46.307689 27781 tablet_copy_service.cc:342] P 6331f77dae9d452184e8cd0cbf5f1b43: Request end of tablet copy session 91d11eae5ef6432ca86eb8ed446b6ea6-3aa09c722530431384e5e7ee9759230c received from {username='slave'} at 127.21.84.132:59727
I20240501 08:12:46.307790 27781 tablet_copy_service.cc:433] P 6331f77dae9d452184e8cd0cbf5f1b43: ending tablet copy session 91d11eae5ef6432ca86eb8ed446b6ea6-3aa09c722530431384e5e7ee9759230c on tablet 3aa09c722530431384e5e7ee9759230c with peer 91d11eae5ef6432ca86eb8ed446b6ea6
I20240501 08:12:46.301630 27760 raft_consensus.cc:2932] T 30130709805e4fa6ae94a46034710863 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:46.314182 1791 ts_tablet_manager.cc:1875] T 132d1b9866574dcea3beed5835e4a22f P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:46.329090 1734 tablet_bootstrap.cc:492] T cb6e886e8cdd451d9a192a771482728e P aad81047009a4ffcbc450b60030583b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:46.330842 1734 tablet_bootstrap.cc:492] T cb6e886e8cdd451d9a192a771482728e P aad81047009a4ffcbc450b60030583b5: Bootstrap complete.
I20240501 08:12:46.331138 1734 ts_tablet_manager.cc:1367] T cb6e886e8cdd451d9a192a771482728e P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.159s user 0.007s sys 0.002s
I20240501 08:12:46.331828 1734 raft_consensus.cc:348] T cb6e886e8cdd451d9a192a771482728e P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:12:46.332088 1734 raft_consensus.cc:729] T cb6e886e8cdd451d9a192a771482728e P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: LEARNER
I20240501 08:12:46.341732 1791 ts_tablet_manager.cc:1888] T 132d1b9866574dcea3beed5835e4a22f P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:46.341951 1791 log.cc:1196] T 132d1b9866574dcea3beed5835e4a22f P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/132d1b9866574dcea3beed5835e4a22f
I20240501 08:12:46.357050 28149 raft_consensus.cc:1203] T 3aa09c722530431384e5e7ee9759230c P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:46.332307 1734 consensus_queue.cc:260] T cb6e886e8cdd451d9a192a771482728e P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:12:46.359899 1734 ts_tablet_manager.cc:1398] T cb6e886e8cdd451d9a192a771482728e P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20240501 08:12:46.362056 30693 tablet_copy_service.cc:342] P 9a022e1a501f4d44a0468054d6c3de47: Request end of tablet copy session aad81047009a4ffcbc450b60030583b5-cb6e886e8cdd451d9a192a771482728e received from {username='slave'} at 127.21.84.131:44127
I20240501 08:12:46.362236 30693 tablet_copy_service.cc:433] P 9a022e1a501f4d44a0468054d6c3de47: ending tablet copy session aad81047009a4ffcbc450b60030583b5-cb6e886e8cdd451d9a192a771482728e on tablet cb6e886e8cdd451d9a192a771482728e with peer aad81047009a4ffcbc450b60030583b5
I20240501 08:12:46.390100 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet 132d1b9866574dcea3beed5835e4a22f (table rebalance_test_table_1 [id=0ee45bbb25734f82b00abd2da25300d1]) successfully deleted
I20240501 08:12:46.431026 1768 tablet_bootstrap.cc:492] T f2baff8656804ce2bfb64a9280e16855 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:46.431686 1768 tablet_bootstrap.cc:492] T f2baff8656804ce2bfb64a9280e16855 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap complete.
I20240501 08:12:46.432996 1768 ts_tablet_manager.cc:1367] T f2baff8656804ce2bfb64a9280e16855 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.161s user 0.037s sys 0.015s
I20240501 08:12:46.435528 1768 raft_consensus.cc:348] T f2baff8656804ce2bfb64a9280e16855 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:12:46.435937 1768 raft_consensus.cc:729] T f2baff8656804ce2bfb64a9280e16855 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: LEARNER
I20240501 08:12:46.436247 1768 consensus_queue.cc:260] T f2baff8656804ce2bfb64a9280e16855 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:12:46.451007 1768 ts_tablet_manager.cc:1398] T f2baff8656804ce2bfb64a9280e16855 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240501 08:12:46.456192 27781 tablet_copy_service.cc:342] P 6331f77dae9d452184e8cd0cbf5f1b43: Request end of tablet copy session 9a022e1a501f4d44a0468054d6c3de47-f2baff8656804ce2bfb64a9280e16855 received from {username='slave'} at 127.21.84.134:48929
I20240501 08:12:46.456367 27781 tablet_copy_service.cc:433] P 6331f77dae9d452184e8cd0cbf5f1b43: ending tablet copy session 9a022e1a501f4d44a0468054d6c3de47-f2baff8656804ce2bfb64a9280e16855 on tablet f2baff8656804ce2bfb64a9280e16855 with peer 9a022e1a501f4d44a0468054d6c3de47
I20240501 08:12:46.479245 27760 consensus_queue.cc:237] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:12:46.481053 30800 raft_consensus.cc:1261] T e3d07789edc74dec975e35e9d94cb0ef P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:46.481604 1089 consensus_queue.cc:1035] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:46.482151 27891 raft_consensus.cc:1261] T e3d07789edc74dec975e35e9d94cb0ef P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:46.482841 1744 consensus_queue.cc:1035] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:46.485070 1744 raft_consensus.cc:2932] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:46.490394 27891 raft_consensus.cc:2932] T e3d07789edc74dec975e35e9d94cb0ef P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:46.494881 1582 tablet_bootstrap.cc:492] T ac4914417d9e44e7a9591be5386f1d8b P 31811708c6774037b82df44024b229b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:46.495241 30800 raft_consensus.cc:2932] T e3d07789edc74dec975e35e9d94cb0ef P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:46.500483 27612 catalog_manager.cc:5484] T e3d07789edc74dec975e35e9d94cb0ef P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index 5 to 6, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:46.502162 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e3d07789edc74dec975e35e9d94cb0ef with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:46.495330 1582 tablet_bootstrap.cc:492] T ac4914417d9e44e7a9591be5386f1d8b P 31811708c6774037b82df44024b229b6: Bootstrap complete.
I20240501 08:12:46.503285 1582 ts_tablet_manager.cc:1367] T ac4914417d9e44e7a9591be5386f1d8b P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.355s user 0.045s sys 0.017s
I20240501 08:12:46.506011 28131 tablet_service.cc:1514] Processing DeleteTablet for tablet e3d07789edc74dec975e35e9d94cb0ef with delete_type TABLET_DATA_TOMBSTONED (TS 91d11eae5ef6432ca86eb8ed446b6ea6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:46.531785 1582 raft_consensus.cc:348] T ac4914417d9e44e7a9591be5386f1d8b P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
I20240501 08:12:46.532135 1582 raft_consensus.cc:729] T ac4914417d9e44e7a9591be5386f1d8b P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: LEARNER
I20240501 08:12:46.532387 1582 consensus_queue.cc:260] T ac4914417d9e44e7a9591be5386f1d8b P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
I20240501 08:12:46.533003 1582 ts_tablet_manager.cc:1398] T ac4914417d9e44e7a9591be5386f1d8b P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.030s user 0.001s sys 0.000s
I20240501 08:12:46.538563 27781 tablet_copy_service.cc:342] P 6331f77dae9d452184e8cd0cbf5f1b43: Request end of tablet copy session 31811708c6774037b82df44024b229b6-ac4914417d9e44e7a9591be5386f1d8b received from {username='slave'} at 127.21.84.133:35347
I20240501 08:12:46.538715 27781 tablet_copy_service.cc:433] P 6331f77dae9d452184e8cd0cbf5f1b43: ending tablet copy session 31811708c6774037b82df44024b229b6-ac4914417d9e44e7a9591be5386f1d8b on tablet ac4914417d9e44e7a9591be5386f1d8b with peer 31811708c6774037b82df44024b229b6
I20240501 08:12:46.555743 1815 tablet_replica.cc:330] T e3d07789edc74dec975e35e9d94cb0ef P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:46.556005 1815 raft_consensus.cc:2229] T e3d07789edc74dec975e35e9d94cb0ef P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:46.556241 1815 raft_consensus.cc:2258] T e3d07789edc74dec975e35e9d94cb0ef P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:46.561812 1815 ts_tablet_manager.cc:1875] T e3d07789edc74dec975e35e9d94cb0ef P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:46.569712 1815 ts_tablet_manager.cc:1888] T e3d07789edc74dec975e35e9d94cb0ef P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:46.569883 1815 log.cc:1196] T e3d07789edc74dec975e35e9d94cb0ef P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/e3d07789edc74dec975e35e9d94cb0ef
I20240501 08:12:46.570724 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet e3d07789edc74dec975e35e9d94cb0ef (table rebalance_test_table_5 [id=b19758166c02438c9fc4cfa835acc028]) successfully deleted
I20240501 08:12:46.629679 30934 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ecab521101074a7b80b862642059daf2"
dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56488
I20240501 08:12:46.629997 30934 raft_consensus.cc:595] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:12:46.641610 28016 raft_consensus.cc:1203] T cb6e886e8cdd451d9a192a771482728e P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:46.658265 28149 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "98c8e3a6f66a42b08507668f7c3a8af1"
dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41860
I20240501 08:12:46.658567 28149 raft_consensus.cc:595] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:12:46.679818 28149 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "30130709805e4fa6ae94a46034710863"
dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41870
I20240501 08:12:46.680055 28149 raft_consensus.cc:595] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:12:46.737139 30672 raft_consensus.cc:1203] T f2baff8656804ce2bfb64a9280e16855 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:46.751986 1735 raft_consensus.cc:979] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6: : Instructing follower aad81047009a4ffcbc450b60030583b5 to start an election
I20240501 08:12:46.752180 1735 raft_consensus.cc:1067] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Signalling peer aad81047009a4ffcbc450b60030583b5 to start an election
I20240501 08:12:46.753470 28018 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "30130709805e4fa6ae94a46034710863"
dest_uuid: "aad81047009a4ffcbc450b60030583b5"
from {username='slave'} at 127.21.84.132:57283
I20240501 08:12:46.753711 28018 raft_consensus.cc:482] T 30130709805e4fa6ae94a46034710863 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:46.753799 28018 raft_consensus.cc:3037] T 30130709805e4fa6ae94a46034710863 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:46.756705 28018 raft_consensus.cc:504] T 30130709805e4fa6ae94a46034710863 P aad81047009a4ffcbc450b60030583b5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:12:46.758092 27760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30130709805e4fa6ae94a46034710863" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:12:46.758316 27760 raft_consensus.cc:3037] T 30130709805e4fa6ae94a46034710863 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:46.759675 28149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30130709805e4fa6ae94a46034710863" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:12:46.759860 28149 raft_consensus.cc:3032] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:12:46.759941 28149 raft_consensus.cc:729] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:12:46.760260 28149 consensus_queue.cc:260] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:12:46.760661 28149 raft_consensus.cc:3037] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:46.761073 27760 raft_consensus.cc:2454] T 30130709805e4fa6ae94a46034710863 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 2.
I20240501 08:12:46.762542 28018 leader_election.cc:288] T 30130709805e4fa6ae94a46034710863 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 2 election: Requested vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:46.763092 30800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30130709805e4fa6ae94a46034710863" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:46.763340 30800 raft_consensus.cc:3037] T 30130709805e4fa6ae94a46034710863 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:46.763352 28149 raft_consensus.cc:2454] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 2.
I20240501 08:12:46.763765 27955 leader_election.cc:302] T 30130709805e4fa6ae94a46034710863 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 91d11eae5ef6432ca86eb8ed446b6ea6, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:12:46.764113 877 raft_consensus.cc:2781] T 30130709805e4fa6ae94a46034710863 P aad81047009a4ffcbc450b60030583b5 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:12:46.764235 877 raft_consensus.cc:686] T 30130709805e4fa6ae94a46034710863 P aad81047009a4ffcbc450b60030583b5 [term 2 LEADER]: Becoming Leader. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Running, Role: LEADER
I20240501 08:12:46.764510 877 consensus_queue.cc:237] T 30130709805e4fa6ae94a46034710863 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:12:46.768075 30934 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c0e30c62c2d441618195f30f15647487"
dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56498
I20240501 08:12:46.768330 30934 raft_consensus.cc:595] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:12:46.768787 30536 raft_consensus.cc:1203] T ac4914417d9e44e7a9591be5386f1d8b P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:46.772300 30800 raft_consensus.cc:2454] T 30130709805e4fa6ae94a46034710863 P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 2.
I20240501 08:12:46.773749 27612 catalog_manager.cc:5484] T 30130709805e4fa6ae94a46034710863 P aad81047009a4ffcbc450b60030583b5 reported cstate change: term changed from 1 to 2, leader changed from 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) to aad81047009a4ffcbc450b60030583b5 (127.21.84.131). New cstate: current_term: 2 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:46.792083 1379 raft_consensus.cc:1050] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11: attempting to promote NON_VOTER f1525cbc82a24d13b65f136695c903af to VOTER
I20240501 08:12:46.792887 1379 consensus_queue.cc:237] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:46.808210 1378 raft_consensus.cc:979] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11: : Instructing follower f1525cbc82a24d13b65f136695c903af to start an election
I20240501 08:12:46.808418 1378 raft_consensus.cc:1067] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Signalling peer f1525cbc82a24d13b65f136695c903af to start an election
I20240501 08:12:46.809283 27891 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ecab521101074a7b80b862642059daf2"
dest_uuid: "f1525cbc82a24d13b65f136695c903af"
from {username='slave'} at 127.21.84.136:54747
I20240501 08:12:46.809566 27891 raft_consensus.cc:482] T ecab521101074a7b80b862642059daf2 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:46.809672 27891 raft_consensus.cc:3037] T ecab521101074a7b80b862642059daf2 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:46.811414 27890 raft_consensus.cc:1261] T b605254185d140d8ab90fcaa5229f168 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:46.812062 30672 raft_consensus.cc:1261] T b605254185d140d8ab90fcaa5229f168 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:46.812427 1378 consensus_queue.cc:1035] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:46.812645 28149 raft_consensus.cc:1261] T b605254185d140d8ab90fcaa5229f168 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:46.812621 27891 raft_consensus.cc:504] T ecab521101074a7b80b862642059daf2 P f1525cbc82a24d13b65f136695c903af [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:46.813628 1378 consensus_queue.cc:1035] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:46.813673 30934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecab521101074a7b80b862642059daf2" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:46.813896 30934 raft_consensus.cc:3032] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:12:46.814014 30934 raft_consensus.cc:729] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:12:46.814205 1378 consensus_queue.cc:1035] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:46.814239 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecab521101074a7b80b862642059daf2" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:46.814286 27891 leader_election.cc:288] T ecab521101074a7b80b862642059daf2 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 2 election: Requested vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:46.814435 30536 raft_consensus.cc:3037] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:46.814390 30934 consensus_queue.cc:260] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:46.814788 30934 raft_consensus.cc:3037] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:46.815069 27760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecab521101074a7b80b862642059daf2" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:12:46.815244 27760 raft_consensus.cc:3037] T ecab521101074a7b80b862642059daf2 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:46.817173 30536 raft_consensus.cc:2454] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 2.
I20240501 08:12:46.817576 30934 raft_consensus.cc:2454] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 2.
I20240501 08:12:46.818162 27827 leader_election.cc:302] T ecab521101074a7b80b862642059daf2 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:46.818483 1711 raft_consensus.cc:2781] T ecab521101074a7b80b862642059daf2 P f1525cbc82a24d13b65f136695c903af [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:12:46.818605 1711 raft_consensus.cc:686] T ecab521101074a7b80b862642059daf2 P f1525cbc82a24d13b65f136695c903af [term 2 LEADER]: Becoming Leader. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Running, Role: LEADER
I20240501 08:12:46.818887 1711 consensus_queue.cc:237] T ecab521101074a7b80b862642059daf2 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:46.831739 27612 catalog_manager.cc:5484] T ecab521101074a7b80b862642059daf2 P f1525cbc82a24d13b65f136695c903af reported cstate change: term changed from 1 to 2, leader changed from a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) to f1525cbc82a24d13b65f136695c903af (127.21.84.130). New cstate: current_term: 2 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:46.834267 27760 raft_consensus.cc:2454] T ecab521101074a7b80b862642059daf2 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 2.
I20240501 08:12:46.838047 1738 raft_consensus.cc:979] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6: : Instructing follower f1525cbc82a24d13b65f136695c903af to start an election
I20240501 08:12:46.838228 1738 raft_consensus.cc:1067] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Signalling peer f1525cbc82a24d13b65f136695c903af to start an election
I20240501 08:12:46.840029 27892 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "98c8e3a6f66a42b08507668f7c3a8af1"
dest_uuid: "f1525cbc82a24d13b65f136695c903af"
from {username='slave'} at 127.21.84.132:35465
I20240501 08:12:46.840245 27892 raft_consensus.cc:482] T 98c8e3a6f66a42b08507668f7c3a8af1 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:46.840335 27892 raft_consensus.cc:3037] T 98c8e3a6f66a42b08507668f7c3a8af1 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:46.843462 27892 raft_consensus.cc:504] T 98c8e3a6f66a42b08507668f7c3a8af1 P f1525cbc82a24d13b65f136695c903af [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:12:46.846648 27892 leader_election.cc:288] T 98c8e3a6f66a42b08507668f7c3a8af1 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 2 election: Requested vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:46.846993 30935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c8e3a6f66a42b08507668f7c3a8af1" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:46.847267 30935 raft_consensus.cc:3037] T 98c8e3a6f66a42b08507668f7c3a8af1 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:46.850389 30800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c8e3a6f66a42b08507668f7c3a8af1" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:46.850697 30800 raft_consensus.cc:3037] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:46.850888 28149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c8e3a6f66a42b08507668f7c3a8af1" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:12:46.851121 28149 raft_consensus.cc:3032] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:12:46.851205 28149 raft_consensus.cc:729] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:12:46.851466 28149 consensus_queue.cc:260] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:12:46.851900 28149 raft_consensus.cc:3037] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:46.851716 1378 raft_consensus.cc:2932] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:46.854620 30672 raft_consensus.cc:2932] T b605254185d140d8ab90fcaa5229f168 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:46.857753 30935 raft_consensus.cc:2454] T 98c8e3a6f66a42b08507668f7c3a8af1 P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 2.
I20240501 08:12:46.859043 28149 raft_consensus.cc:2454] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 2.
I20240501 08:12:46.859531 27825 leader_election.cc:302] T 98c8e3a6f66a42b08507668f7c3a8af1 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, a29cd590999b4f32849ff5689cd53c11, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:46.860265 1712 raft_consensus.cc:2781] T 98c8e3a6f66a42b08507668f7c3a8af1 P f1525cbc82a24d13b65f136695c903af [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:12:46.860388 1712 raft_consensus.cc:686] T 98c8e3a6f66a42b08507668f7c3a8af1 P f1525cbc82a24d13b65f136695c903af [term 2 LEADER]: Becoming Leader. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Running, Role: LEADER
I20240501 08:12:46.860615 1712 consensus_queue.cc:237] T 98c8e3a6f66a42b08507668f7c3a8af1 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:12:46.862035 30800 raft_consensus.cc:2454] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 2.
I20240501 08:12:46.867122 27612 catalog_manager.cc:5484] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:46.867494 28149 raft_consensus.cc:2932] T b605254185d140d8ab90fcaa5229f168 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:46.879057 27892 raft_consensus.cc:2932] T b605254185d140d8ab90fcaa5229f168 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:46.889477 27612 catalog_manager.cc:5484] T 98c8e3a6f66a42b08507668f7c3a8af1 P f1525cbc82a24d13b65f136695c903af reported cstate change: term changed from 1 to 2, leader changed from 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) to f1525cbc82a24d13b65f136695c903af (127.21.84.130). New cstate: current_term: 2 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:46.911134 1378 raft_consensus.cc:979] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11: : Instructing follower 9a022e1a501f4d44a0468054d6c3de47 to start an election
I20240501 08:12:46.911301 1378 raft_consensus.cc:1067] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Signalling peer 9a022e1a501f4d44a0468054d6c3de47 to start an election
I20240501 08:12:46.911988 30669 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c0e30c62c2d441618195f30f15647487"
dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
from {username='slave'} at 127.21.84.136:58465
I20240501 08:12:46.912278 30669 raft_consensus.cc:482] T c0e30c62c2d441618195f30f15647487 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:46.912405 30669 raft_consensus.cc:3037] T c0e30c62c2d441618195f30f15647487 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:46.916616 30669 raft_consensus.cc:504] T c0e30c62c2d441618195f30f15647487 P 9a022e1a501f4d44a0468054d6c3de47 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:46.918005 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e30c62c2d441618195f30f15647487" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:12:46.918006 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e30c62c2d441618195f30f15647487" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:12:46.918205 27890 raft_consensus.cc:3037] T c0e30c62c2d441618195f30f15647487 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:46.918203 27759 raft_consensus.cc:3037] T c0e30c62c2d441618195f30f15647487 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:46.920892 27890 raft_consensus.cc:2454] T c0e30c62c2d441618195f30f15647487 P f1525cbc82a24d13b65f136695c903af [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 2.
I20240501 08:12:46.920892 27759 raft_consensus.cc:2454] T c0e30c62c2d441618195f30f15647487 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 2.
I20240501 08:12:46.921923 30935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e30c62c2d441618195f30f15647487" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:46.922230 30935 raft_consensus.cc:3032] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:12:46.922317 30935 raft_consensus.cc:729] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:12:46.922717 30608 leader_election.cc:302] T c0e30c62c2d441618195f30f15647487 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 9a022e1a501f4d44a0468054d6c3de47, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:46.922613 30935 consensus_queue.cc:260] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:46.923041 1679 raft_consensus.cc:2781] T c0e30c62c2d441618195f30f15647487 P 9a022e1a501f4d44a0468054d6c3de47 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:12:46.923036 30935 raft_consensus.cc:3037] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:46.923197 1679 raft_consensus.cc:686] T c0e30c62c2d441618195f30f15647487 P 9a022e1a501f4d44a0468054d6c3de47 [term 2 LEADER]: Becoming Leader. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Running, Role: LEADER
I20240501 08:12:46.923498 1679 consensus_queue.cc:237] T c0e30c62c2d441618195f30f15647487 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:46.924002 30669 leader_election.cc:288] T c0e30c62c2d441618195f30f15647487 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 election: Requested vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:12:46.925504 30935 raft_consensus.cc:2454] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 2.
I20240501 08:12:46.928776 27612 catalog_manager.cc:5484] T c0e30c62c2d441618195f30f15647487 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: term changed from 1 to 2, leader changed from a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) to 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134). New cstate: current_term: 2 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:46.949191 1426 raft_consensus.cc:1050] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43: attempting to promote NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 to VOTER
I20240501 08:12:46.949927 1426 consensus_queue.cc:237] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:12:46.956077 28149 raft_consensus.cc:1261] T 3aa09c722530431384e5e7ee9759230c P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:46.956276 30536 raft_consensus.cc:1261] T 3aa09c722530431384e5e7ee9759230c P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:46.956315 30803 raft_consensus.cc:1261] T 3aa09c722530431384e5e7ee9759230c P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:46.956921 1744 consensus_queue.cc:1035] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:46.957226 1089 consensus_queue.cc:1035] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:46.957417 1766 consensus_queue.cc:1035] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:46.965962 1089 raft_consensus.cc:2932] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:46.969825 30536 raft_consensus.cc:2932] T 3aa09c722530431384e5e7ee9759230c P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:46.975194 30803 raft_consensus.cc:2932] T 3aa09c722530431384e5e7ee9759230c P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:46.976071 28149 raft_consensus.cc:2932] T 3aa09c722530431384e5e7ee9759230c P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:46.978549 27612 catalog_manager.cc:5484] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:46.989539 27759 consensus_queue.cc:237] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:12:46.991307 28149 raft_consensus.cc:1261] T 3aa09c722530431384e5e7ee9759230c P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:46.991775 1766 consensus_queue.cc:1035] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:46.993702 30537 raft_consensus.cc:1261] T 3aa09c722530431384e5e7ee9759230c P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:46.994643 1766 consensus_queue.cc:1035] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:46.995879 1426 raft_consensus.cc:2932] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:46.996616 28151 raft_consensus.cc:2932] T 3aa09c722530431384e5e7ee9759230c P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:46.998159 30537 raft_consensus.cc:2932] T 3aa09c722530431384e5e7ee9759230c P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:47.001688 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3aa09c722530431384e5e7ee9759230c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:47.005013 27616 catalog_manager.cc:5484] T 3aa09c722530431384e5e7ee9759230c P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index 3 to 4, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:47.010123 30784 tablet_service.cc:1514] Processing DeleteTablet for tablet 3aa09c722530431384e5e7ee9759230c with delete_type TABLET_DATA_TOMBSTONED (TS 64998323a1f4482d8704bf20594d13a5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38952
I20240501 08:12:47.010875 1915 tablet_replica.cc:330] T 3aa09c722530431384e5e7ee9759230c P 64998323a1f4482d8704bf20594d13a5: stopping tablet replica
I20240501 08:12:47.011217 1915 raft_consensus.cc:2229] T 3aa09c722530431384e5e7ee9759230c P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:47.013610 1915 raft_consensus.cc:2258] T 3aa09c722530431384e5e7ee9759230c P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:47.015053 1915 ts_tablet_manager.cc:1875] T 3aa09c722530431384e5e7ee9759230c P 64998323a1f4482d8704bf20594d13a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:47.021564 1915 ts_tablet_manager.cc:1888] T 3aa09c722530431384e5e7ee9759230c P 64998323a1f4482d8704bf20594d13a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:47.021853 1915 log.cc:1196] T 3aa09c722530431384e5e7ee9759230c P 64998323a1f4482d8704bf20594d13a5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal/wals/3aa09c722530431384e5e7ee9759230c
I20240501 08:12:47.023149 27605 catalog_manager.cc:4830] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): tablet 3aa09c722530431384e5e7ee9759230c (table rebalancer_extra_table_6 [id=2b1f439c36b24b19bec52d25077c330d]) successfully deleted
I20240501 08:12:47.067746 30935 consensus_queue.cc:237] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
I20240501 08:12:47.070262 28151 raft_consensus.cc:1261] T a1965bd993d748f6a33cf93e8a3d70bf P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:47.070252 30669 raft_consensus.cc:1261] T a1965bd993d748f6a33cf93e8a3d70bf P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:12:47.070487 30873 consensus_peers.cc:485] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 -> Peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): Couldn't send request to peer 31811708c6774037b82df44024b229b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a1965bd993d748f6a33cf93e8a3d70bf. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:47.070971 1378 consensus_queue.cc:1035] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:47.071484 1378 consensus_queue.cc:1035] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:47.077683 1378 raft_consensus.cc:2932] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:12:47.078404 30672 raft_consensus.cc:2932] T a1965bd993d748f6a33cf93e8a3d70bf P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:12:47.084831 27616 catalog_manager.cc:5484] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index -1 to 2, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:47.097426 28151 raft_consensus.cc:2932] T a1965bd993d748f6a33cf93e8a3d70bf P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:12:47.125191 30803 consensus_queue.cc:237] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:12:47.127410 30672 raft_consensus.cc:1261] T 40a4b09831f84433b6dc126b8445cf41 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:47.127494 27759 raft_consensus.cc:1261] T 40a4b09831f84433b6dc126b8445cf41 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:12:47.127789 30739 consensus_peers.cc:485] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 -> Peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Couldn't send request to peer aad81047009a4ffcbc450b60030583b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 40a4b09831f84433b6dc126b8445cf41. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:47.128154 1892 consensus_queue.cc:1035] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:47.128484 1335 consensus_queue.cc:1035] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:47.131163 1335 raft_consensus.cc:2932] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:47.131875 27759 raft_consensus.cc:2932] T 40a4b09831f84433b6dc126b8445cf41 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:47.135151 30672 raft_consensus.cc:2932] T 40a4b09831f84433b6dc126b8445cf41 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:47.137553 27618 catalog_manager.cc:5484] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:47.147397 30803 raft_consensus.cc:1261] T 30130709805e4fa6ae94a46034710863 P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:12:47.147982 1772 consensus_queue.cc:1035] T 30130709805e4fa6ae94a46034710863 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:47.162360 30935 consensus_queue.cc:237] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:12:47.164944 30536 raft_consensus.cc:1261] T 8277d70d49f54197af60e0be86d34a9b P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:12:47.165113 30870 consensus_peers.cc:485] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 -> Peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Couldn't send request to peer 9a022e1a501f4d44a0468054d6c3de47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8277d70d49f54197af60e0be86d34a9b. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:47.165228 30803 raft_consensus.cc:1261] T 8277d70d49f54197af60e0be86d34a9b P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:47.165515 1940 consensus_queue.cc:1035] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:47.165830 1730 consensus_queue.cc:1035] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:47.168586 1940 raft_consensus.cc:2932] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:47.169693 30803 raft_consensus.cc:2932] T 8277d70d49f54197af60e0be86d34a9b P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:47.172505 30536 raft_consensus.cc:2932] T 8277d70d49f54197af60e0be86d34a9b P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:47.178524 27618 catalog_manager.cc:5484] T 8277d70d49f54197af60e0be86d34a9b P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index -1 to 2, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:47.194543 27760 raft_consensus.cc:1261] T 30130709805e4fa6ae94a46034710863 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:12:47.195238 30935 consensus_queue.cc:237] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:12:47.196813 1772 consensus_queue.cc:1035] T 30130709805e4fa6ae94a46034710863 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:47.199225 30536 raft_consensus.cc:1261] T 82d6e5e399744b96a66373240ce1764d P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:12:47.199589 30870 consensus_peers.cc:485] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 -> Peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Couldn't send request to peer 91d11eae5ef6432ca86eb8ed446b6ea6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 82d6e5e399744b96a66373240ce1764d. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:47.199923 1378 consensus_queue.cc:1035] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:47.200083 27760 raft_consensus.cc:1261] T 82d6e5e399744b96a66373240ce1764d P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:47.200598 1378 consensus_queue.cc:1035] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:47.206022 1426 raft_consensus.cc:1050] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43: attempting to promote NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 to VOTER
I20240501 08:12:47.207134 1426 consensus_queue.cc:237] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:12:47.215312 1378 raft_consensus.cc:2932] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:12:47.217510 28151 raft_consensus.cc:1261] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:12:47.218629 27758 raft_consensus.cc:2932] T 82d6e5e399744b96a66373240ce1764d P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:12:47.219259 1772 consensus_queue.cc:1035] T 30130709805e4fa6ae94a46034710863 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:47.224079 27616 catalog_manager.cc:5484] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:47.228104 28018 consensus_queue.cc:237] T 30130709805e4fa6ae94a46034710863 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:12:47.234695 30536 raft_consensus.cc:2932] T 82d6e5e399744b96a66373240ce1764d P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:12:47.295137 27759 raft_consensus.cc:1261] T ecab521101074a7b80b862642059daf2 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:12:47.295980 1679 raft_consensus.cc:1050] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47: attempting to promote NON_VOTER aad81047009a4ffcbc450b60030583b5 to VOTER
I20240501 08:12:47.296026 1712 consensus_queue.cc:1035] T ecab521101074a7b80b862642059daf2 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:47.296867 1679 consensus_queue.cc:237] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } }
I20240501 08:12:47.298611 30672 raft_consensus.cc:1261] T f2baff8656804ce2bfb64a9280e16855 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:47.299434 1426 consensus_queue.cc:1035] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:47.299866 28149 raft_consensus.cc:1261] T f2baff8656804ce2bfb64a9280e16855 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:47.301635 1089 consensus_queue.cc:1035] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:47.302300 30935 raft_consensus.cc:1261] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:12:47.303678 1712 consensus_queue.cc:1035] T ecab521101074a7b80b862642059daf2 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:47.311928 1089 raft_consensus.cc:1050] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43: attempting to promote NON_VOTER 31811708c6774037b82df44024b229b6 to VOTER
I20240501 08:12:47.312767 1089 consensus_queue.cc:237] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } }
I20240501 08:12:47.313627 30800 raft_consensus.cc:1261] T 30130709805e4fa6ae94a46034710863 P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:12:47.315338 1975 consensus_queue.cc:1035] T 30130709805e4fa6ae94a46034710863 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:47.320468 877 raft_consensus.cc:2932] T 30130709805e4fa6ae94a46034710863 P aad81047009a4ffcbc450b60030583b5 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:47.332253 27758 raft_consensus.cc:1261] T 30130709805e4fa6ae94a46034710863 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:12:47.333007 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 30130709805e4fa6ae94a46034710863 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:47.332784 30800 raft_consensus.cc:2932] T 30130709805e4fa6ae94a46034710863 P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:47.338711 27616 catalog_manager.cc:5484] T 30130709805e4fa6ae94a46034710863 P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 5 to 7, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New cstate: current_term: 2 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:47.349915 28131 tablet_service.cc:1514] Processing DeleteTablet for tablet 30130709805e4fa6ae94a46034710863 with delete_type TABLET_DATA_TOMBSTONED (TS 91d11eae5ef6432ca86eb8ed446b6ea6 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:47.350571 28151 raft_consensus.cc:1261] T cb6e886e8cdd451d9a192a771482728e P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:47.352428 30536 raft_consensus.cc:1261] T cb6e886e8cdd451d9a192a771482728e P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:47.353077 1974 consensus_queue.cc:1035] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:47.356159 1975 consensus_queue.cc:1035] T 30130709805e4fa6ae94a46034710863 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:47.357854 27758 raft_consensus.cc:2932] T 30130709805e4fa6ae94a46034710863 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:47.362358 1679 consensus_queue.cc:1035] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:47.365476 28018 raft_consensus.cc:1261] T cb6e886e8cdd451d9a192a771482728e P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:47.366372 1970 consensus_queue.cc:1035] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:47.381616 1766 raft_consensus.cc:2932] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:47.384302 30803 raft_consensus.cc:1261] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:12:47.385023 1712 consensus_queue.cc:1035] T 98c8e3a6f66a42b08507668f7c3a8af1 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:47.386404 28149 raft_consensus.cc:2932] T f2baff8656804ce2bfb64a9280e16855 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:47.388485 1986 tablet_replica.cc:330] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:47.389384 1986 raft_consensus.cc:2229] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:47.390204 1986 raft_consensus.cc:2258] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:47.392201 30537 raft_consensus.cc:1261] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:12:47.393052 1716 consensus_queue.cc:1035] T ecab521101074a7b80b862642059daf2 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:47.399103 30669 raft_consensus.cc:2932] T f2baff8656804ce2bfb64a9280e16855 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
W20240501 08:12:47.409758 27958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.130:36503, user_credentials={real_user=slave}} blocked reactor thread for 145417us
I20240501 08:12:47.415627 27759 raft_consensus.cc:1261] T c0e30c62c2d441618195f30f15647487 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:12:47.415313 1683 raft_consensus.cc:2932] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:47.415596 28149 raft_consensus.cc:2932] T cb6e886e8cdd451d9a192a771482728e P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:47.425654 1986 ts_tablet_manager.cc:1875] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:47.427214 1679 consensus_queue.cc:1035] T c0e30c62c2d441618195f30f15647487 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:47.428258 30934 raft_consensus.cc:1261] T 98c8e3a6f66a42b08507668f7c3a8af1 P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:12:47.429829 1716 consensus_queue.cc:1035] T 98c8e3a6f66a42b08507668f7c3a8af1 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:47.432437 28018 raft_consensus.cc:2932] T cb6e886e8cdd451d9a192a771482728e P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:47.458834 28151 raft_consensus.cc:1261] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:12:47.460772 1711 consensus_queue.cc:1035] T 98c8e3a6f66a42b08507668f7c3a8af1 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:47.474192 28016 raft_consensus.cc:1261] T f2baff8656804ce2bfb64a9280e16855 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:47.488247 30936 raft_consensus.cc:1261] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:12:47.488821 1683 consensus_queue.cc:1035] T c0e30c62c2d441618195f30f15647487 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:47.491641 27619 catalog_manager.cc:5484] T f2baff8656804ce2bfb64a9280e16855 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 4 to 5, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:47.503998 27619 catalog_manager.cc:5484] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:47.510349 1426 consensus_queue.cc:1035] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:47.512343 27890 raft_consensus.cc:1261] T c0e30c62c2d441618195f30f15647487 P f1525cbc82a24d13b65f136695c903af [term 2 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:12:47.544631 30670 consensus_queue.cc:237] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } }
I20240501 08:12:47.547389 28016 raft_consensus.cc:1261] T cb6e886e8cdd451d9a192a771482728e P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:47.548274 1679 consensus_queue.cc:1035] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:47.551312 28151 raft_consensus.cc:1261] T cb6e886e8cdd451d9a192a771482728e P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:47.551435 1683 consensus_queue.cc:1035] T c0e30c62c2d441618195f30f15647487 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:47.552402 1683 consensus_queue.cc:1035] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:47.552469 30536 raft_consensus.cc:2932] T cb6e886e8cdd451d9a192a771482728e P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:47.552897 1683 raft_consensus.cc:2932] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:47.563097 28149 raft_consensus.cc:2932] T cb6e886e8cdd451d9a192a771482728e P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:47.575302 28149 consensus_queue.cc:237] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
I20240501 08:12:47.576449 27604 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cb6e886e8cdd451d9a192a771482728e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:47.578950 27615 catalog_manager.cc:5484] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New cstate: current_term: 1 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:47.583065 30514 tablet_service.cc:1514] Processing DeleteTablet for tablet cb6e886e8cdd451d9a192a771482728e with delete_type TABLET_DATA_TOMBSTONED (TS 31811708c6774037b82df44024b229b6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45078
I20240501 08:12:47.585340 2004 tablet_replica.cc:330] T cb6e886e8cdd451d9a192a771482728e P 31811708c6774037b82df44024b229b6: stopping tablet replica
I20240501 08:12:47.585752 2004 raft_consensus.cc:2229] T cb6e886e8cdd451d9a192a771482728e P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:47.586220 2004 raft_consensus.cc:2258] T cb6e886e8cdd451d9a192a771482728e P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:47.588579 2004 ts_tablet_manager.cc:1875] T cb6e886e8cdd451d9a192a771482728e P 31811708c6774037b82df44024b229b6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:47.600163 28016 raft_consensus.cc:2932] T cb6e886e8cdd451d9a192a771482728e P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
W20240501 08:12:47.601094 28088 consensus_peers.cc:485] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 -> Peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): Couldn't send request to peer 31811708c6774037b82df44024b229b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bc5778711a4c479881088af790f5b7d5. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:47.602524 27891 raft_consensus.cc:1261] T bc5778711a4c479881088af790f5b7d5 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:47.603365 1735 consensus_queue.cc:1035] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:47.607019 28018 raft_consensus.cc:1261] T bc5778711a4c479881088af790f5b7d5 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:47.612133 1735 consensus_queue.cc:1035] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:47.623880 30537 raft_consensus.cc:1261] T ac4914417d9e44e7a9591be5386f1d8b P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:47.624912 1766 consensus_queue.cc:1035] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:47.625342 2007 ts_tablet_manager.cc:902] T a1965bd993d748f6a33cf93e8a3d70bf P 31811708c6774037b82df44024b229b6: Initiating tablet copy from peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:47.633510 28018 raft_consensus.cc:2932] T f2baff8656804ce2bfb64a9280e16855 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:47.639763 28022 raft_consensus.cc:1261] T ac4914417d9e44e7a9591be5386f1d8b P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:47.645298 2012 ts_tablet_manager.cc:902] T 40a4b09831f84433b6dc126b8445cf41 P aad81047009a4ffcbc450b60030583b5: Initiating tablet copy from peer 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:47.662061 1766 consensus_queue.cc:1035] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:47.664330 1740 raft_consensus.cc:2932] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:12:47.677404 27890 raft_consensus.cc:1261] T ac4914417d9e44e7a9591be5386f1d8b P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:47.677762 27892 consensus_queue.cc:237] T 98c8e3a6f66a42b08507668f7c3a8af1 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:12:47.678715 27611 catalog_manager.cc:5484] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: config changed from index -1 to 4, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:47.681704 1744 consensus_queue.cc:1035] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:47.682432 30800 raft_consensus.cc:1261] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:12:47.683069 1711 consensus_queue.cc:1035] T 98c8e3a6f66a42b08507668f7c3a8af1 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:47.686676 30934 raft_consensus.cc:1261] T 98c8e3a6f66a42b08507668f7c3a8af1 P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:12:47.688216 1711 raft_consensus.cc:2932] T 98c8e3a6f66a42b08507668f7c3a8af1 P f1525cbc82a24d13b65f136695c903af [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:47.695286 30800 raft_consensus.cc:2932] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:47.699658 1712 consensus_queue.cc:1035] T 98c8e3a6f66a42b08507668f7c3a8af1 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:47.702777 30934 raft_consensus.cc:2932] T 98c8e3a6f66a42b08507668f7c3a8af1 P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:47.712083 27611 catalog_manager.cc:5484] T 98c8e3a6f66a42b08507668f7c3a8af1 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index 3 to 5, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New cstate: current_term: 2 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:47.719956 27607 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 98c8e3a6f66a42b08507668f7c3a8af1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:47.734553 2025 ts_tablet_manager.cc:902] T 8277d70d49f54197af60e0be86d34a9b P 9a022e1a501f4d44a0468054d6c3de47: Initiating tablet copy from peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:47.746116 30669 consensus_queue.cc:237] T c0e30c62c2d441618195f30f15647487 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:47.749251 27759 raft_consensus.cc:1261] T c0e30c62c2d441618195f30f15647487 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:12:47.750108 1683 consensus_queue.cc:1035] T c0e30c62c2d441618195f30f15647487 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:47.752847 1679 raft_consensus.cc:2932] T c0e30c62c2d441618195f30f15647487 P 9a022e1a501f4d44a0468054d6c3de47 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:47.759745 27604 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c0e30c62c2d441618195f30f15647487 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:47.763739 27759 raft_consensus.cc:2932] T c0e30c62c2d441618195f30f15647487 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:47.765291 27611 catalog_manager.cc:5484] T c0e30c62c2d441618195f30f15647487 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 3 to 5, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New cstate: current_term: 2 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:47.652722 1986 ts_tablet_manager.cc:1888] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 08:12:47.770756 1986 log.cc:1196] T 30130709805e4fa6ae94a46034710863 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/30130709805e4fa6ae94a46034710863
I20240501 08:12:47.771685 30916 tablet_service.cc:1514] Processing DeleteTablet for tablet c0e30c62c2d441618195f30f15647487 with delete_type TABLET_DATA_TOMBSTONED (TS a29cd590999b4f32849ff5689cd53c11 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54036
I20240501 08:12:47.778241 27891 raft_consensus.cc:1261] T c0e30c62c2d441618195f30f15647487 P f1525cbc82a24d13b65f136695c903af [term 2 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:12:47.778935 1976 consensus_queue.cc:1035] T c0e30c62c2d441618195f30f15647487 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:47.780870 27889 raft_consensus.cc:2932] T c0e30c62c2d441618195f30f15647487 P f1525cbc82a24d13b65f136695c903af [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:47.791561 2030 tablet_replica.cc:330] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11: stopping tablet replica
I20240501 08:12:47.795037 2030 raft_consensus.cc:2229] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:47.796806 2030 raft_consensus.cc:2258] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:47.799854 27891 raft_consensus.cc:2932] T bc5778711a4c479881088af790f5b7d5 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:12:47.800235 1766 raft_consensus.cc:2932] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:47.798535 2030 ts_tablet_manager.cc:1875] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:47.801473 28016 raft_consensus.cc:2932] T ac4914417d9e44e7a9591be5386f1d8b P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:47.804904 30536 raft_consensus.cc:2932] T ac4914417d9e44e7a9591be5386f1d8b P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:47.809441 27615 catalog_manager.cc:5484] T ac4914417d9e44e7a9591be5386f1d8b P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 4 to 5, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:47.809775 28022 raft_consensus.cc:2932] T bc5778711a4c479881088af790f5b7d5 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:12:47.810150 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet 30130709805e4fa6ae94a46034710863 (table rebalance_test_table_1 [id=0ee45bbb25734f82b00abd2da25300d1]) successfully deleted
I20240501 08:12:47.814791 2004 ts_tablet_manager.cc:1888] T cb6e886e8cdd451d9a192a771482728e P 31811708c6774037b82df44024b229b6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:47.814952 2004 log.cc:1196] T cb6e886e8cdd451d9a192a771482728e P 31811708c6774037b82df44024b229b6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/wals/cb6e886e8cdd451d9a192a771482728e
I20240501 08:12:47.815276 28131 tablet_service.cc:1514] Processing DeleteTablet for tablet 98c8e3a6f66a42b08507668f7c3a8af1 with delete_type TABLET_DATA_TOMBSTONED (TS 91d11eae5ef6432ca86eb8ed446b6ea6 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:47.815749 27607 catalog_manager.cc:4830] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): tablet cb6e886e8cdd451d9a192a771482728e (table rebalancer_extra_table_38 [id=099418eaa97444bf9763310daefd6f13]) successfully deleted
W20240501 08:12:47.817917 27826 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.135:33503, user_credentials={real_user=slave}} blocked reactor thread for 117806us
I20240501 08:12:47.819185 1986 tablet_replica.cc:330] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:47.819499 1986 raft_consensus.cc:2229] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:47.819794 1986 raft_consensus.cc:2258] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:47.821202 1986 ts_tablet_manager.cc:1875] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:47.828239 27758 consensus_queue.cc:237] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } }
I20240501 08:12:47.831236 30536 raft_consensus.cc:1261] T ac4914417d9e44e7a9591be5386f1d8b P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:47.831805 1744 consensus_queue.cc:1035] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:47.834795 1766 raft_consensus.cc:2932] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:47.838553 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ac4914417d9e44e7a9591be5386f1d8b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:47.843079 27892 consensus_queue.cc:237] T ecab521101074a7b80b862642059daf2 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:47.843097 27620 catalog_manager.cc:5484] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:47.847208 30537 raft_consensus.cc:1261] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:12:47.847620 27759 raft_consensus.cc:1261] T ecab521101074a7b80b862642059daf2 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:12:47.848202 1712 consensus_queue.cc:1035] T ecab521101074a7b80b862642059daf2 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:47.848718 1711 consensus_queue.cc:1035] T ecab521101074a7b80b862642059daf2 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:47.848883 28000 tablet_service.cc:1514] Processing DeleteTablet for tablet ac4914417d9e44e7a9591be5386f1d8b with delete_type TABLET_DATA_TOMBSTONED (TS aad81047009a4ffcbc450b60030583b5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:47.848872 30537 raft_consensus.cc:2932] T ac4914417d9e44e7a9591be5386f1d8b P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:47.852630 1712 raft_consensus.cc:2932] T ecab521101074a7b80b862642059daf2 P f1525cbc82a24d13b65f136695c903af [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:47.860334 27607 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ecab521101074a7b80b862642059daf2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:47.862820 30537 raft_consensus.cc:2932] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:47.866784 27612 catalog_manager.cc:5484] T ecab521101074a7b80b862642059daf2 P f1525cbc82a24d13b65f136695c903af reported cstate change: config changed from index 3 to 5, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New cstate: current_term: 2 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:47.867000 27758 raft_consensus.cc:2932] T ecab521101074a7b80b862642059daf2 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:47.870682 30916 tablet_service.cc:1514] Processing DeleteTablet for tablet ecab521101074a7b80b862642059daf2 with delete_type TABLET_DATA_TOMBSTONED (TS a29cd590999b4f32849ff5689cd53c11 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54036
I20240501 08:12:47.873543 2012 tablet_copy_client.cc:308] T 40a4b09831f84433b6dc126b8445cf41 P aad81047009a4ffcbc450b60030583b5: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.135:33503
I20240501 08:12:47.874720 30825 tablet_copy_service.cc:140] P 64998323a1f4482d8704bf20594d13a5: Received BeginTabletCopySession request for tablet 40a4b09831f84433b6dc126b8445cf41 from peer aad81047009a4ffcbc450b60030583b5 ({username='slave'} at 127.21.84.131:43681)
I20240501 08:12:47.874898 30825 tablet_copy_service.cc:161] P 64998323a1f4482d8704bf20594d13a5: Beginning new tablet copy session on tablet 40a4b09831f84433b6dc126b8445cf41 from peer aad81047009a4ffcbc450b60030583b5 at {username='slave'} at 127.21.84.131:43681: session id = aad81047009a4ffcbc450b60030583b5-40a4b09831f84433b6dc126b8445cf41
I20240501 08:12:47.877758 30825 tablet_copy_source_session.cc:205] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:47.878232 27889 raft_consensus.cc:1261] T ac4914417d9e44e7a9591be5386f1d8b P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:47.878790 27889 raft_consensus.cc:2932] T ac4914417d9e44e7a9591be5386f1d8b P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:47.879401 1986 ts_tablet_manager.cc:1888] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240501 08:12:47.879568 1986 log.cc:1196] T 98c8e3a6f66a42b08507668f7c3a8af1 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/98c8e3a6f66a42b08507668f7c3a8af1
I20240501 08:12:47.880425 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet 98c8e3a6f66a42b08507668f7c3a8af1 (table rebalancer_extra_table_18 [id=c906ec3019644e00bd7c63c8bcc3d323]) successfully deleted
I20240501 08:12:47.884513 2032 ts_tablet_manager.cc:902] T 82d6e5e399744b96a66373240ce1764d P 91d11eae5ef6432ca86eb8ed446b6ea6: Initiating tablet copy from peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:47.887014 2012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40a4b09831f84433b6dc126b8445cf41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:47.881687 2030 ts_tablet_manager.cc:1888] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240501 08:12:47.888113 2025 tablet_copy_client.cc:308] T 8277d70d49f54197af60e0be86d34a9b P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.136:46845
I20240501 08:12:47.888319 2030 log.cc:1196] T c0e30c62c2d441618195f30f15647487 P a29cd590999b4f32849ff5689cd53c11: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/wals/c0e30c62c2d441618195f30f15647487
I20240501 08:12:47.889737 27606 catalog_manager.cc:4830] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): tablet c0e30c62c2d441618195f30f15647487 (table rebalancer_extra_table_32 [id=62ce67b20f6e4a2b80c594fcbe8f131e]) successfully deleted
I20240501 08:12:47.890087 1426 consensus_queue.cc:1035] T ac4914417d9e44e7a9591be5386f1d8b P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:47.890276 2030 tablet_replica.cc:330] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11: stopping tablet replica
I20240501 08:12:47.890501 2030 raft_consensus.cc:2229] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:47.890722 2030 raft_consensus.cc:2258] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:47.890980 30957 tablet_copy_service.cc:140] P a29cd590999b4f32849ff5689cd53c11: Received BeginTabletCopySession request for tablet 8277d70d49f54197af60e0be86d34a9b from peer 9a022e1a501f4d44a0468054d6c3de47 ({username='slave'} at 127.21.84.134:36153)
I20240501 08:12:47.891129 30957 tablet_copy_service.cc:161] P a29cd590999b4f32849ff5689cd53c11: Beginning new tablet copy session on tablet 8277d70d49f54197af60e0be86d34a9b from peer 9a022e1a501f4d44a0468054d6c3de47 at {username='slave'} at 127.21.84.134:36153: session id = 9a022e1a501f4d44a0468054d6c3de47-8277d70d49f54197af60e0be86d34a9b
I20240501 08:12:47.894405 2035 tablet_replica.cc:330] T ac4914417d9e44e7a9591be5386f1d8b P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:47.894651 2035 raft_consensus.cc:2229] T ac4914417d9e44e7a9591be5386f1d8b P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:47.894954 2035 raft_consensus.cc:2258] T ac4914417d9e44e7a9591be5386f1d8b P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:47.896696 30957 tablet_copy_source_session.cc:205] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:47.897178 27889 raft_consensus.cc:2932] T ac4914417d9e44e7a9591be5386f1d8b P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:47.901176 2025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8277d70d49f54197af60e0be86d34a9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:47.904824 2035 ts_tablet_manager.cc:1875] T ac4914417d9e44e7a9591be5386f1d8b P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:47.908489 2007 tablet_copy_client.cc:308] T a1965bd993d748f6a33cf93e8a3d70bf P 31811708c6774037b82df44024b229b6: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.136:46845
I20240501 08:12:47.909279 30957 tablet_copy_service.cc:140] P a29cd590999b4f32849ff5689cd53c11: Received BeginTabletCopySession request for tablet a1965bd993d748f6a33cf93e8a3d70bf from peer 31811708c6774037b82df44024b229b6 ({username='slave'} at 127.21.84.133:34677)
I20240501 08:12:47.909540 30957 tablet_copy_service.cc:161] P a29cd590999b4f32849ff5689cd53c11: Beginning new tablet copy session on tablet a1965bd993d748f6a33cf93e8a3d70bf from peer 31811708c6774037b82df44024b229b6 at {username='slave'} at 127.21.84.133:34677: session id = 31811708c6774037b82df44024b229b6-a1965bd993d748f6a33cf93e8a3d70bf
I20240501 08:12:47.912861 30957 tablet_copy_source_session.cc:205] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:47.914985 2007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1965bd993d748f6a33cf93e8a3d70bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:47.921857 2030 ts_tablet_manager.cc:1875] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:47.925736 2007 tablet_copy_client.cc:783] T a1965bd993d748f6a33cf93e8a3d70bf P 31811708c6774037b82df44024b229b6: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:47.926075 2007 tablet_copy_client.cc:647] T a1965bd993d748f6a33cf93e8a3d70bf P 31811708c6774037b82df44024b229b6: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:47.927879 2007 tablet_copy_client.cc:523] T a1965bd993d748f6a33cf93e8a3d70bf P 31811708c6774037b82df44024b229b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:47.931108 2012 tablet_copy_client.cc:783] T 40a4b09831f84433b6dc126b8445cf41 P aad81047009a4ffcbc450b60030583b5: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:47.932694 2012 tablet_copy_client.cc:647] T 40a4b09831f84433b6dc126b8445cf41 P aad81047009a4ffcbc450b60030583b5: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:47.932768 2007 tablet_bootstrap.cc:492] T a1965bd993d748f6a33cf93e8a3d70bf P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:47.936426 2012 tablet_copy_client.cc:523] T 40a4b09831f84433b6dc126b8445cf41 P aad81047009a4ffcbc450b60030583b5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:47.946002 2032 tablet_copy_client.cc:308] T 82d6e5e399744b96a66373240ce1764d P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.136:46845
I20240501 08:12:47.946307 2035 ts_tablet_manager.cc:1888] T ac4914417d9e44e7a9591be5386f1d8b P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:47.946722 2035 log.cc:1196] T ac4914417d9e44e7a9591be5386f1d8b P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/ac4914417d9e44e7a9591be5386f1d8b
I20240501 08:12:47.951094 30957 tablet_copy_service.cc:140] P a29cd590999b4f32849ff5689cd53c11: Received BeginTabletCopySession request for tablet 82d6e5e399744b96a66373240ce1764d from peer 91d11eae5ef6432ca86eb8ed446b6ea6 ({username='slave'} at 127.21.84.132:58509)
I20240501 08:12:47.951282 30957 tablet_copy_service.cc:161] P a29cd590999b4f32849ff5689cd53c11: Beginning new tablet copy session on tablet 82d6e5e399744b96a66373240ce1764d from peer 91d11eae5ef6432ca86eb8ed446b6ea6 at {username='slave'} at 127.21.84.132:58509: session id = 91d11eae5ef6432ca86eb8ed446b6ea6-82d6e5e399744b96a66373240ce1764d
I20240501 08:12:47.951393 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet ac4914417d9e44e7a9591be5386f1d8b (table rebalance_test_table_3 [id=56fa27c3d3de4793aadbcaf54cdb0ac5]) successfully deleted
I20240501 08:12:47.951835 2025 tablet_copy_client.cc:783] T 8277d70d49f54197af60e0be86d34a9b P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:47.952219 2025 tablet_copy_client.cc:647] T 8277d70d49f54197af60e0be86d34a9b P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:47.947993 2012 tablet_bootstrap.cc:492] T 40a4b09831f84433b6dc126b8445cf41 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:47.954272 2025 tablet_copy_client.cc:523] T 8277d70d49f54197af60e0be86d34a9b P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:47.954497 30957 tablet_copy_source_session.cc:205] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:47.956482 2032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82d6e5e399744b96a66373240ce1764d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:47.961203 2007 tablet_bootstrap.cc:492] T a1965bd993d748f6a33cf93e8a3d70bf P 31811708c6774037b82df44024b229b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:47.961798 2007 tablet_bootstrap.cc:492] T a1965bd993d748f6a33cf93e8a3d70bf P 31811708c6774037b82df44024b229b6: Bootstrap complete.
I20240501 08:12:47.962100 2012 tablet_bootstrap.cc:492] T 40a4b09831f84433b6dc126b8445cf41 P aad81047009a4ffcbc450b60030583b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:47.962206 2007 ts_tablet_manager.cc:1367] T a1965bd993d748f6a33cf93e8a3d70bf P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.006s
I20240501 08:12:47.962669 2012 tablet_bootstrap.cc:492] T 40a4b09831f84433b6dc126b8445cf41 P aad81047009a4ffcbc450b60030583b5: Bootstrap complete.
I20240501 08:12:47.962976 2012 ts_tablet_manager.cc:1367] T 40a4b09831f84433b6dc126b8445cf41 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s
I20240501 08:12:47.963747 2012 raft_consensus.cc:348] T 40a4b09831f84433b6dc126b8445cf41 P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:12:47.964079 2012 raft_consensus.cc:729] T 40a4b09831f84433b6dc126b8445cf41 P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: LEARNER
I20240501 08:12:47.964156 2007 raft_consensus.cc:348] T a1965bd993d748f6a33cf93e8a3d70bf P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
I20240501 08:12:47.964516 2007 raft_consensus.cc:729] T a1965bd993d748f6a33cf93e8a3d70bf P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: LEARNER
I20240501 08:12:47.964325 2012 consensus_queue.cc:260] T 40a4b09831f84433b6dc126b8445cf41 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:12:47.964756 2007 consensus_queue.cc:260] T a1965bd993d748f6a33cf93e8a3d70bf P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
I20240501 08:12:47.967769 2007 ts_tablet_manager.cc:1398] T a1965bd993d748f6a33cf93e8a3d70bf P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:12:47.968458 30957 tablet_copy_service.cc:342] P a29cd590999b4f32849ff5689cd53c11: Request end of tablet copy session 31811708c6774037b82df44024b229b6-a1965bd993d748f6a33cf93e8a3d70bf received from {username='slave'} at 127.21.84.133:34677
I20240501 08:12:47.968619 30957 tablet_copy_service.cc:433] P a29cd590999b4f32849ff5689cd53c11: ending tablet copy session 31811708c6774037b82df44024b229b6-a1965bd993d748f6a33cf93e8a3d70bf on tablet a1965bd993d748f6a33cf93e8a3d70bf with peer 31811708c6774037b82df44024b229b6
I20240501 08:12:47.971848 2012 ts_tablet_manager.cc:1398] T 40a4b09831f84433b6dc126b8445cf41 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240501 08:12:47.978363 2032 tablet_copy_client.cc:783] T 82d6e5e399744b96a66373240ce1764d P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:47.978658 2032 tablet_copy_client.cc:647] T 82d6e5e399744b96a66373240ce1764d P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:47.985850 2025 tablet_bootstrap.cc:492] T 8277d70d49f54197af60e0be86d34a9b P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:47.986613 30825 tablet_copy_service.cc:342] P 64998323a1f4482d8704bf20594d13a5: Request end of tablet copy session aad81047009a4ffcbc450b60030583b5-40a4b09831f84433b6dc126b8445cf41 received from {username='slave'} at 127.21.84.131:43681
I20240501 08:12:47.986768 30825 tablet_copy_service.cc:433] P 64998323a1f4482d8704bf20594d13a5: ending tablet copy session aad81047009a4ffcbc450b60030583b5-40a4b09831f84433b6dc126b8445cf41 on tablet 40a4b09831f84433b6dc126b8445cf41 with peer aad81047009a4ffcbc450b60030583b5
I20240501 08:12:47.986754 2032 tablet_copy_client.cc:523] T 82d6e5e399744b96a66373240ce1764d P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:47.990571 2032 tablet_bootstrap.cc:492] T 82d6e5e399744b96a66373240ce1764d P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:48.009004 2030 ts_tablet_manager.cc:1888] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240501 08:12:48.009303 2030 log.cc:1196] T ecab521101074a7b80b862642059daf2 P a29cd590999b4f32849ff5689cd53c11: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/wals/ecab521101074a7b80b862642059daf2
I20240501 08:12:48.012421 2025 tablet_bootstrap.cc:492] T 8277d70d49f54197af60e0be86d34a9b P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:48.013164 2025 tablet_bootstrap.cc:492] T 8277d70d49f54197af60e0be86d34a9b P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap complete.
I20240501 08:12:48.013574 2025 ts_tablet_manager.cc:1367] T 8277d70d49f54197af60e0be86d34a9b P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.010s
I20240501 08:12:48.014146 27606 catalog_manager.cc:4830] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): tablet ecab521101074a7b80b862642059daf2 (table rebalancer_extra_table_0 [id=d6a1f8dc8c864cafb8fa47bf4e8e7b3d]) successfully deleted
I20240501 08:12:48.014889 2025 raft_consensus.cc:348] T 8277d70d49f54197af60e0be86d34a9b P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:12:48.018146 2025 raft_consensus.cc:729] T 8277d70d49f54197af60e0be86d34a9b P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: LEARNER
I20240501 08:12:48.018564 2025 consensus_queue.cc:260] T 8277d70d49f54197af60e0be86d34a9b P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:12:48.034395 2025 ts_tablet_manager.cc:1398] T 8277d70d49f54197af60e0be86d34a9b P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.021s user 0.002s sys 0.001s
I20240501 08:12:48.036178 28151 consensus_queue.cc:237] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:12:48.040158 2032 tablet_bootstrap.cc:492] T 82d6e5e399744b96a66373240ce1764d P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:48.040621 2032 tablet_bootstrap.cc:492] T 82d6e5e399744b96a66373240ce1764d P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap complete.
I20240501 08:12:48.041417 30957 tablet_copy_service.cc:342] P a29cd590999b4f32849ff5689cd53c11: Request end of tablet copy session 9a022e1a501f4d44a0468054d6c3de47-8277d70d49f54197af60e0be86d34a9b received from {username='slave'} at 127.21.84.134:36153
I20240501 08:12:48.041556 30957 tablet_copy_service.cc:433] P a29cd590999b4f32849ff5689cd53c11: ending tablet copy session 9a022e1a501f4d44a0468054d6c3de47-8277d70d49f54197af60e0be86d34a9b on tablet 8277d70d49f54197af60e0be86d34a9b with peer 9a022e1a501f4d44a0468054d6c3de47
I20240501 08:12:48.050702 2032 ts_tablet_manager.cc:1367] T 82d6e5e399744b96a66373240ce1764d P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.060s user 0.018s sys 0.026s
I20240501 08:12:48.051425 2032 raft_consensus.cc:348] T 82d6e5e399744b96a66373240ce1764d P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:12:48.051667 2032 raft_consensus.cc:729] T 82d6e5e399744b96a66373240ce1764d P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: LEARNER
I20240501 08:12:48.051841 2032 consensus_queue.cc:260] T 82d6e5e399744b96a66373240ce1764d P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:12:48.052374 2032 ts_tablet_manager.cc:1398] T 82d6e5e399744b96a66373240ce1764d P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:48.058882 30957 tablet_copy_service.cc:342] P a29cd590999b4f32849ff5689cd53c11: Request end of tablet copy session 91d11eae5ef6432ca86eb8ed446b6ea6-82d6e5e399744b96a66373240ce1764d received from {username='slave'} at 127.21.84.132:58509
I20240501 08:12:48.059032 30957 tablet_copy_service.cc:433] P a29cd590999b4f32849ff5689cd53c11: ending tablet copy session 91d11eae5ef6432ca86eb8ed446b6ea6-82d6e5e399744b96a66373240ce1764d on tablet 82d6e5e399744b96a66373240ce1764d with peer 91d11eae5ef6432ca86eb8ed446b6ea6
I20240501 08:12:48.063251 30536 raft_consensus.cc:1261] T 7c3b2b31a9e747c69ff67969d878bc64 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:48.065644 30936 raft_consensus.cc:1261] T 7c3b2b31a9e747c69ff67969d878bc64 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:48.069171 1735 consensus_queue.cc:1035] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:48.070245 28020 raft_consensus.cc:1203] T 40a4b09831f84433b6dc126b8445cf41 P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240501 08:12:48.075757 28087 consensus_peers.cc:485] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6 -> Peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Couldn't send request to peer aad81047009a4ffcbc450b60030583b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c3b2b31a9e747c69ff67969d878bc64. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:48.076071 1740 consensus_queue.cc:1035] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:48.090973 1735 raft_consensus.cc:2932] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:48.099190 30934 raft_consensus.cc:2932] T 7c3b2b31a9e747c69ff67969d878bc64 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:48.115208 27611 catalog_manager.cc:5484] T 7c3b2b31a9e747c69ff67969d878bc64 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:48.149422 30536 raft_consensus.cc:2932] T 7c3b2b31a9e747c69ff67969d878bc64 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:48.165843 30800 consensus_queue.cc:237] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:48.169121 30669 raft_consensus.cc:1261] T 6c9bb7ab554e4023892339cd9e0c2a75 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:12:48.169225 30740 consensus_peers.cc:485] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c9bb7ab554e4023892339cd9e0c2a75. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:48.169710 2013 consensus_queue.cc:1035] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:48.171996 30934 raft_consensus.cc:1261] T 6c9bb7ab554e4023892339cd9e0c2a75 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:48.172777 2013 consensus_queue.cc:1035] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:48.175452 1593 raft_consensus.cc:2932] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:48.177570 30536 raft_consensus.cc:1203] T a1965bd993d748f6a33cf93e8a3d70bf P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:48.178503 30934 raft_consensus.cc:2932] T 6c9bb7ab554e4023892339cd9e0c2a75 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:48.190853 30672 raft_consensus.cc:2932] T 6c9bb7ab554e4023892339cd9e0c2a75 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:48.191103 27612 catalog_manager.cc:5484] T 6c9bb7ab554e4023892339cd9e0c2a75 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:48.205785 2007 ts_tablet_manager.cc:902] T bc5778711a4c479881088af790f5b7d5 P 31811708c6774037b82df44024b229b6: Initiating tablet copy from peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:48.210952 2007 tablet_copy_client.cc:308] T bc5778711a4c479881088af790f5b7d5 P 31811708c6774037b82df44024b229b6: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.132:41323
I20240501 08:12:48.211796 28172 tablet_copy_service.cc:140] P 91d11eae5ef6432ca86eb8ed446b6ea6: Received BeginTabletCopySession request for tablet bc5778711a4c479881088af790f5b7d5 from peer 31811708c6774037b82df44024b229b6 ({username='slave'} at 127.21.84.133:45793)
I20240501 08:12:48.211985 28172 tablet_copy_service.cc:161] P 91d11eae5ef6432ca86eb8ed446b6ea6: Beginning new tablet copy session on tablet bc5778711a4c479881088af790f5b7d5 from peer 31811708c6774037b82df44024b229b6 at {username='slave'} at 127.21.84.133:45793: session id = 31811708c6774037b82df44024b229b6-bc5778711a4c479881088af790f5b7d5
I20240501 08:12:48.214891 28172 tablet_copy_source_session.cc:205] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:48.216323 2007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc5778711a4c479881088af790f5b7d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:48.222522 30800 consensus_queue.cc:237] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:12:48.222750 2007 tablet_copy_client.cc:783] T bc5778711a4c479881088af790f5b7d5 P 31811708c6774037b82df44024b229b6: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:48.223009 2007 tablet_copy_client.cc:647] T bc5778711a4c479881088af790f5b7d5 P 31811708c6774037b82df44024b229b6: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:48.224682 2007 tablet_copy_client.cc:523] T bc5778711a4c479881088af790f5b7d5 P 31811708c6774037b82df44024b229b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:48.226539 30669 raft_consensus.cc:1261] T 325b3dbba80d491dab81840e8eb171e9 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:48.227759 2062 consensus_queue.cc:1035] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:48.228863 28151 raft_consensus.cc:1261] T 325b3dbba80d491dab81840e8eb171e9 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:48.229533 2062 consensus_queue.cc:1035] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:12:48.230158 30739 consensus_peers.cc:485] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 325b3dbba80d491dab81840e8eb171e9. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:48.230625 2007 tablet_bootstrap.cc:492] T bc5778711a4c479881088af790f5b7d5 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:12:48.235155 2062 raft_consensus.cc:2932] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:48.240891 30669 raft_consensus.cc:2932] T 325b3dbba80d491dab81840e8eb171e9 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:48.241389 28151 raft_consensus.cc:2932] T 325b3dbba80d491dab81840e8eb171e9 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:48.248903 27611 catalog_manager.cc:5484] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:48.305092 30669 raft_consensus.cc:1203] T 8277d70d49f54197af60e0be86d34a9b P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:48.323628 30800 consensus_queue.cc:237] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
W20240501 08:12:48.326462 30739 consensus_peers.cc:485] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 -> Peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): Couldn't send request to peer a29cd590999b4f32849ff5689cd53c11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d3673fe03fb64c47b30dc2b79042f44a. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:48.327088 30537 raft_consensus.cc:1261] T d3673fe03fb64c47b30dc2b79042f44a P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:48.327852 2062 consensus_queue.cc:1035] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:48.330617 28151 raft_consensus.cc:1261] T d3673fe03fb64c47b30dc2b79042f44a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:48.331359 2062 consensus_queue.cc:1035] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:48.332801 2007 tablet_bootstrap.cc:492] T bc5778711a4c479881088af790f5b7d5 P 31811708c6774037b82df44024b229b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:48.333261 2007 tablet_bootstrap.cc:492] T bc5778711a4c479881088af790f5b7d5 P 31811708c6774037b82df44024b229b6: Bootstrap complete.
I20240501 08:12:48.333577 2007 ts_tablet_manager.cc:1367] T bc5778711a4c479881088af790f5b7d5 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.103s user 0.055s sys 0.004s
I20240501 08:12:48.334627 2007 raft_consensus.cc:348] T bc5778711a4c479881088af790f5b7d5 P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
I20240501 08:12:48.335000 2007 raft_consensus.cc:729] T bc5778711a4c479881088af790f5b7d5 P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: LEARNER
I20240501 08:12:48.335222 2007 consensus_queue.cc:260] T bc5778711a4c479881088af790f5b7d5 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
I20240501 08:12:48.334970 1892 raft_consensus.cc:2932] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:48.339404 28151 raft_consensus.cc:2932] T d3673fe03fb64c47b30dc2b79042f44a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:48.341719 2007 ts_tablet_manager.cc:1398] T bc5778711a4c479881088af790f5b7d5 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240501 08:12:48.343348 28172 tablet_copy_service.cc:342] P 91d11eae5ef6432ca86eb8ed446b6ea6: Request end of tablet copy session 31811708c6774037b82df44024b229b6-bc5778711a4c479881088af790f5b7d5 received from {username='slave'} at 127.21.84.133:45793
I20240501 08:12:48.343478 28172 tablet_copy_service.cc:433] P 91d11eae5ef6432ca86eb8ed446b6ea6: ending tablet copy session 31811708c6774037b82df44024b229b6-bc5778711a4c479881088af790f5b7d5 on tablet bc5778711a4c479881088af790f5b7d5 with peer 31811708c6774037b82df44024b229b6
I20240501 08:12:48.343683 30537 raft_consensus.cc:2932] T d3673fe03fb64c47b30dc2b79042f44a P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:48.349177 27612 catalog_manager.cc:5484] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index -1 to 2, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:48.439483 28151 raft_consensus.cc:1203] T 82d6e5e399744b96a66373240ce1764d P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:48.443872 27889 consensus_queue.cc:237] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:12:48.456036 30537 raft_consensus.cc:1261] T 0e06cf862c954e8fb1ca087ec5184497 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:48.456178 30800 raft_consensus.cc:1261] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:48.456725 1712 consensus_queue.cc:1035] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:12:48.457715 27827 consensus_peers.cc:485] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af -> Peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Couldn't send request to peer aad81047009a4ffcbc450b60030583b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e06cf862c954e8fb1ca087ec5184497. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:48.458616 1593 raft_consensus.cc:1050] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5: attempting to promote NON_VOTER aad81047009a4ffcbc450b60030583b5 to VOTER
I20240501 08:12:48.459587 1593 consensus_queue.cc:237] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } }
I20240501 08:12:48.466300 1712 consensus_queue.cc:1035] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:48.471093 2021 raft_consensus.cc:2932] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:48.471487 30800 raft_consensus.cc:2932] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:48.479457 27620 catalog_manager.cc:5484] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:48.503051 27759 raft_consensus.cc:1261] T 40a4b09831f84433b6dc126b8445cf41 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:48.503093 30672 raft_consensus.cc:1261] T 40a4b09831f84433b6dc126b8445cf41 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:48.503801 2062 consensus_queue.cc:1035] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:48.504338 1892 consensus_queue.cc:1035] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:48.506126 28016 raft_consensus.cc:1261] T 40a4b09831f84433b6dc126b8445cf41 P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:48.507054 1892 consensus_queue.cc:1035] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:48.523054 30537 raft_consensus.cc:2932] T 0e06cf862c954e8fb1ca087ec5184497 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:48.523907 1892 raft_consensus.cc:2932] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:48.526104 28016 raft_consensus.cc:2932] T 40a4b09831f84433b6dc126b8445cf41 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:48.530001 27758 raft_consensus.cc:2932] T 40a4b09831f84433b6dc126b8445cf41 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:48.533789 27620 catalog_manager.cc:5484] T 40a4b09831f84433b6dc126b8445cf41 P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:48.544320 30672 raft_consensus.cc:2932] T 40a4b09831f84433b6dc126b8445cf41 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:48.571290 30803 consensus_queue.cc:237] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } }
I20240501 08:12:48.576278 28022 raft_consensus.cc:1261] T 40a4b09831f84433b6dc126b8445cf41 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:48.577342 1892 consensus_queue.cc:1035] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:48.578796 30672 raft_consensus.cc:1261] T 40a4b09831f84433b6dc126b8445cf41 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:48.579356 1892 consensus_queue.cc:1035] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:48.581964 1593 raft_consensus.cc:2932] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:48.583634 28022 raft_consensus.cc:2932] T 40a4b09831f84433b6dc126b8445cf41 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:48.585775 27605 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 40a4b09831f84433b6dc126b8445cf41 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:48.592563 30672 raft_consensus.cc:2932] T 40a4b09831f84433b6dc126b8445cf41 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:48.593276 27620 catalog_manager.cc:5484] T 40a4b09831f84433b6dc126b8445cf41 P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 3 to 4, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:48.598719 27741 tablet_service.cc:1514] Processing DeleteTablet for tablet 40a4b09831f84433b6dc126b8445cf41 with delete_type TABLET_DATA_TOMBSTONED (TS 6331f77dae9d452184e8cd0cbf5f1b43 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:48.604990 2099 ts_tablet_manager.cc:902] T 7c3b2b31a9e747c69ff67969d878bc64 P aad81047009a4ffcbc450b60030583b5: Initiating tablet copy from peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:48.631510 2106 ts_tablet_manager.cc:902] T 325b3dbba80d491dab81840e8eb171e9 P 6331f77dae9d452184e8cd0cbf5f1b43: Initiating tablet copy from peer 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:48.634855 2101 tablet_replica.cc:330] T 40a4b09831f84433b6dc126b8445cf41 P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:48.635197 2101 raft_consensus.cc:2229] T 40a4b09831f84433b6dc126b8445cf41 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:48.635541 2101 raft_consensus.cc:2258] T 40a4b09831f84433b6dc126b8445cf41 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:48.636963 2101 ts_tablet_manager.cc:1875] T 40a4b09831f84433b6dc126b8445cf41 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:48.647099 2105 ts_tablet_manager.cc:902] T 6c9bb7ab554e4023892339cd9e0c2a75 P f1525cbc82a24d13b65f136695c903af: Initiating tablet copy from peer 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:48.648696 2101 ts_tablet_manager.cc:1888] T 40a4b09831f84433b6dc126b8445cf41 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:48.648916 2101 log.cc:1196] T 40a4b09831f84433b6dc126b8445cf41 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/40a4b09831f84433b6dc126b8445cf41
I20240501 08:12:48.649837 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet 40a4b09831f84433b6dc126b8445cf41 (table rebalancer_extra_table_32 [id=62ce67b20f6e4a2b80c594fcbe8f131e]) successfully deleted
I20240501 08:12:48.650436 1940 raft_consensus.cc:1050] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11: attempting to promote NON_VOTER 31811708c6774037b82df44024b229b6 to VOTER
I20240501 08:12:48.651304 1940 consensus_queue.cc:237] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } }
I20240501 08:12:48.655144 2106 tablet_copy_client.cc:308] T 325b3dbba80d491dab81840e8eb171e9 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.135:33503
I20240501 08:12:48.656212 30825 tablet_copy_service.cc:140] P 64998323a1f4482d8704bf20594d13a5: Received BeginTabletCopySession request for tablet 325b3dbba80d491dab81840e8eb171e9 from peer 6331f77dae9d452184e8cd0cbf5f1b43 ({username='slave'} at 127.21.84.129:58333)
I20240501 08:12:48.656410 30825 tablet_copy_service.cc:161] P 64998323a1f4482d8704bf20594d13a5: Beginning new tablet copy session on tablet 325b3dbba80d491dab81840e8eb171e9 from peer 6331f77dae9d452184e8cd0cbf5f1b43 at {username='slave'} at 127.21.84.129:58333: session id = 6331f77dae9d452184e8cd0cbf5f1b43-325b3dbba80d491dab81840e8eb171e9
I20240501 08:12:48.656821 30537 raft_consensus.cc:1261] T a1965bd993d748f6a33cf93e8a3d70bf P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:48.657454 1378 consensus_queue.cc:1035] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:48.659037 28151 raft_consensus.cc:1261] T a1965bd993d748f6a33cf93e8a3d70bf P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:48.659912 1378 consensus_queue.cc:1035] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:48.661552 2099 tablet_copy_client.cc:308] T 7c3b2b31a9e747c69ff67969d878bc64 P aad81047009a4ffcbc450b60030583b5: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.132:41323
I20240501 08:12:48.663555 28172 tablet_copy_service.cc:140] P 91d11eae5ef6432ca86eb8ed446b6ea6: Received BeginTabletCopySession request for tablet 7c3b2b31a9e747c69ff67969d878bc64 from peer aad81047009a4ffcbc450b60030583b5 ({username='slave'} at 127.21.84.131:34383)
I20240501 08:12:48.663719 28172 tablet_copy_service.cc:161] P 91d11eae5ef6432ca86eb8ed446b6ea6: Beginning new tablet copy session on tablet 7c3b2b31a9e747c69ff67969d878bc64 from peer aad81047009a4ffcbc450b60030583b5 at {username='slave'} at 127.21.84.131:34383: session id = aad81047009a4ffcbc450b60030583b5-7c3b2b31a9e747c69ff67969d878bc64
I20240501 08:12:48.667886 28172 tablet_copy_source_session.cc:205] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:48.671193 30825 tablet_copy_source_session.cc:205] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:48.672943 2106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 325b3dbba80d491dab81840e8eb171e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:48.677665 2099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c3b2b31a9e747c69ff67969d878bc64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:48.694752 30672 raft_consensus.cc:1261] T a1965bd993d748f6a33cf93e8a3d70bf P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:48.699594 2106 tablet_copy_client.cc:783] T 325b3dbba80d491dab81840e8eb171e9 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:48.700428 2106 tablet_copy_client.cc:647] T 325b3dbba80d491dab81840e8eb171e9 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:48.702358 2106 tablet_copy_client.cc:523] T 325b3dbba80d491dab81840e8eb171e9 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:48.702836 2099 tablet_copy_client.cc:783] T 7c3b2b31a9e747c69ff67969d878bc64 P aad81047009a4ffcbc450b60030583b5: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:48.703056 2099 tablet_copy_client.cc:647] T 7c3b2b31a9e747c69ff67969d878bc64 P aad81047009a4ffcbc450b60030583b5: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:48.707069 1940 consensus_queue.cc:1035] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:48.716326 28151 raft_consensus.cc:2932] T a1965bd993d748f6a33cf93e8a3d70bf P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:48.716969 30534 raft_consensus.cc:1203] T bc5778711a4c479881088af790f5b7d5 P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:48.718744 1940 raft_consensus.cc:2932] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:48.720456 2099 tablet_copy_client.cc:523] T 7c3b2b31a9e747c69ff67969d878bc64 P aad81047009a4ffcbc450b60030583b5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:48.735541 2099 tablet_bootstrap.cc:492] T 7c3b2b31a9e747c69ff67969d878bc64 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:48.735554 2105 tablet_copy_client.cc:308] T 6c9bb7ab554e4023892339cd9e0c2a75 P f1525cbc82a24d13b65f136695c903af: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.135:33503
I20240501 08:12:48.737076 30825 tablet_copy_service.cc:140] P 64998323a1f4482d8704bf20594d13a5: Received BeginTabletCopySession request for tablet 6c9bb7ab554e4023892339cd9e0c2a75 from peer f1525cbc82a24d13b65f136695c903af ({username='slave'} at 127.21.84.130:45079)
I20240501 08:12:48.737255 30825 tablet_copy_service.cc:161] P 64998323a1f4482d8704bf20594d13a5: Beginning new tablet copy session on tablet 6c9bb7ab554e4023892339cd9e0c2a75 from peer f1525cbc82a24d13b65f136695c903af at {username='slave'} at 127.21.84.130:45079: session id = f1525cbc82a24d13b65f136695c903af-6c9bb7ab554e4023892339cd9e0c2a75
I20240501 08:12:48.740216 30825 tablet_copy_source_session.cc:205] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:48.743124 27615 catalog_manager.cc:5484] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index 2 to 3, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:48.743433 30537 raft_consensus.cc:2932] T a1965bd993d748f6a33cf93e8a3d70bf P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:48.747197 2105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c9bb7ab554e4023892339cd9e0c2a75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:48.757392 30934 consensus_queue.cc:237] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } }
I20240501 08:12:48.759025 2105 tablet_copy_client.cc:783] T 6c9bb7ab554e4023892339cd9e0c2a75 P f1525cbc82a24d13b65f136695c903af: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:48.759369 2105 tablet_copy_client.cc:647] T 6c9bb7ab554e4023892339cd9e0c2a75 P f1525cbc82a24d13b65f136695c903af: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:48.761061 2105 tablet_copy_client.cc:523] T 6c9bb7ab554e4023892339cd9e0c2a75 P f1525cbc82a24d13b65f136695c903af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:48.761637 30537 raft_consensus.cc:1261] T a1965bd993d748f6a33cf93e8a3d70bf P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:48.762488 1940 consensus_queue.cc:1035] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:48.763805 28152 raft_consensus.cc:1261] T a1965bd993d748f6a33cf93e8a3d70bf P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:48.764782 1940 consensus_queue.cc:1035] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:48.766001 1940 raft_consensus.cc:2932] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:48.768162 2105 tablet_bootstrap.cc:492] T 6c9bb7ab554e4023892339cd9e0c2a75 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:48.769066 30536 raft_consensus.cc:2932] T a1965bd993d748f6a33cf93e8a3d70bf P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:48.769614 2106 tablet_bootstrap.cc:492] T 325b3dbba80d491dab81840e8eb171e9 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:48.778255 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a1965bd993d748f6a33cf93e8a3d70bf with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:48.780000 27615 catalog_manager.cc:5484] T a1965bd993d748f6a33cf93e8a3d70bf P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:48.787351 28152 raft_consensus.cc:2932] T a1965bd993d748f6a33cf93e8a3d70bf P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:48.794011 30650 tablet_service.cc:1514] Processing DeleteTablet for tablet a1965bd993d748f6a33cf93e8a3d70bf with delete_type TABLET_DATA_TOMBSTONED (TS 9a022e1a501f4d44a0468054d6c3de47 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57156
I20240501 08:12:48.794680 2130 tablet_replica.cc:330] T a1965bd993d748f6a33cf93e8a3d70bf P 9a022e1a501f4d44a0468054d6c3de47: stopping tablet replica
I20240501 08:12:48.795111 2130 raft_consensus.cc:2229] T a1965bd993d748f6a33cf93e8a3d70bf P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:48.795413 2130 pending_rounds.cc:62] T a1965bd993d748f6a33cf93e8a3d70bf P 9a022e1a501f4d44a0468054d6c3de47: Trying to abort 1 pending ops.
I20240501 08:12:48.795538 2130 pending_rounds.cc:68] T a1965bd993d748f6a33cf93e8a3d70bf P 9a022e1a501f4d44a0468054d6c3de47: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7022801586793529344 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "a1965bd993d748f6a33cf93e8a3d70bf" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } } }
I20240501 08:12:48.796126 2130 raft_consensus.cc:2894] T a1965bd993d748f6a33cf93e8a3d70bf P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240501 08:12:48.796308 2130 raft_consensus.cc:2866] T a1965bd993d748f6a33cf93e8a3d70bf P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 08:12:48.796481 2130 raft_consensus.cc:2258] T a1965bd993d748f6a33cf93e8a3d70bf P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:48.799765 2130 ts_tablet_manager.cc:1875] T a1965bd993d748f6a33cf93e8a3d70bf P 9a022e1a501f4d44a0468054d6c3de47: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:48.806663 2130 ts_tablet_manager.cc:1888] T a1965bd993d748f6a33cf93e8a3d70bf P 9a022e1a501f4d44a0468054d6c3de47: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:48.806933 2130 log.cc:1196] T a1965bd993d748f6a33cf93e8a3d70bf P 9a022e1a501f4d44a0468054d6c3de47: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/wals/a1965bd993d748f6a33cf93e8a3d70bf
I20240501 08:12:48.808161 27604 catalog_manager.cc:4830] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): tablet a1965bd993d748f6a33cf93e8a3d70bf (table rebalancer_extra_table_22 [id=dbd065db73d1428c8fe77bd9af3302f6]) successfully deleted
I20240501 08:12:48.811156 2105 tablet_bootstrap.cc:492] T 6c9bb7ab554e4023892339cd9e0c2a75 P f1525cbc82a24d13b65f136695c903af: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:48.814118 2105 tablet_bootstrap.cc:492] T 6c9bb7ab554e4023892339cd9e0c2a75 P f1525cbc82a24d13b65f136695c903af: Bootstrap complete.
I20240501 08:12:48.814368 2105 ts_tablet_manager.cc:1367] T 6c9bb7ab554e4023892339cd9e0c2a75 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.046s user 0.009s sys 0.017s
I20240501 08:12:48.814532 2106 tablet_bootstrap.cc:492] T 325b3dbba80d491dab81840e8eb171e9 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:48.815052 2106 tablet_bootstrap.cc:492] T 325b3dbba80d491dab81840e8eb171e9 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap complete.
I20240501 08:12:48.815016 2105 raft_consensus.cc:348] T 6c9bb7ab554e4023892339cd9e0c2a75 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:48.815230 2105 raft_consensus.cc:729] T 6c9bb7ab554e4023892339cd9e0c2a75 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: LEARNER
I20240501 08:12:48.815305 2106 ts_tablet_manager.cc:1367] T 325b3dbba80d491dab81840e8eb171e9 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.046s user 0.009s sys 0.004s
I20240501 08:12:48.815418 2105 consensus_queue.cc:260] T 6c9bb7ab554e4023892339cd9e0c2a75 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:48.816036 2106 raft_consensus.cc:348] T 325b3dbba80d491dab81840e8eb171e9 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:12:48.816288 2106 raft_consensus.cc:729] T 325b3dbba80d491dab81840e8eb171e9 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: LEARNER
I20240501 08:12:48.816498 2106 consensus_queue.cc:260] T 325b3dbba80d491dab81840e8eb171e9 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:12:48.817097 2106 ts_tablet_manager.cc:1398] T 325b3dbba80d491dab81840e8eb171e9 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:48.817739 2099 tablet_bootstrap.cc:492] T 7c3b2b31a9e747c69ff67969d878bc64 P aad81047009a4ffcbc450b60030583b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:48.817809 30825 tablet_copy_service.cc:342] P 64998323a1f4482d8704bf20594d13a5: Request end of tablet copy session 6331f77dae9d452184e8cd0cbf5f1b43-325b3dbba80d491dab81840e8eb171e9 received from {username='slave'} at 127.21.84.129:58333
I20240501 08:12:48.817994 30825 tablet_copy_service.cc:433] P 64998323a1f4482d8704bf20594d13a5: ending tablet copy session 6331f77dae9d452184e8cd0cbf5f1b43-325b3dbba80d491dab81840e8eb171e9 on tablet 325b3dbba80d491dab81840e8eb171e9 with peer 6331f77dae9d452184e8cd0cbf5f1b43
I20240501 08:12:48.818173 2099 tablet_bootstrap.cc:492] T 7c3b2b31a9e747c69ff67969d878bc64 P aad81047009a4ffcbc450b60030583b5: Bootstrap complete.
I20240501 08:12:48.818409 2099 ts_tablet_manager.cc:1367] T 7c3b2b31a9e747c69ff67969d878bc64 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.083s user 0.000s sys 0.006s
I20240501 08:12:48.819146 2099 raft_consensus.cc:348] T 7c3b2b31a9e747c69ff67969d878bc64 P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:12:48.821564 2099 raft_consensus.cc:729] T 7c3b2b31a9e747c69ff67969d878bc64 P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: LEARNER
I20240501 08:12:48.821779 2105 ts_tablet_manager.cc:1398] T 6c9bb7ab554e4023892339cd9e0c2a75 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240501 08:12:48.821945 2099 consensus_queue.cc:260] T 7c3b2b31a9e747c69ff67969d878bc64 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:12:48.825676 30825 tablet_copy_service.cc:342] P 64998323a1f4482d8704bf20594d13a5: Request end of tablet copy session f1525cbc82a24d13b65f136695c903af-6c9bb7ab554e4023892339cd9e0c2a75 received from {username='slave'} at 127.21.84.130:45079
I20240501 08:12:48.825875 30825 tablet_copy_service.cc:433] P 64998323a1f4482d8704bf20594d13a5: ending tablet copy session f1525cbc82a24d13b65f136695c903af-6c9bb7ab554e4023892339cd9e0c2a75 on tablet 6c9bb7ab554e4023892339cd9e0c2a75 with peer f1525cbc82a24d13b65f136695c903af
I20240501 08:12:48.829476 2099 ts_tablet_manager.cc:1398] T 7c3b2b31a9e747c69ff67969d878bc64 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 08:12:48.830222 28172 tablet_copy_service.cc:342] P 91d11eae5ef6432ca86eb8ed446b6ea6: Request end of tablet copy session aad81047009a4ffcbc450b60030583b5-7c3b2b31a9e747c69ff67969d878bc64 received from {username='slave'} at 127.21.84.131:34383
I20240501 08:12:48.830343 28172 tablet_copy_service.cc:433] P 91d11eae5ef6432ca86eb8ed446b6ea6: ending tablet copy session aad81047009a4ffcbc450b60030583b5-7c3b2b31a9e747c69ff67969d878bc64 on tablet 7c3b2b31a9e747c69ff67969d878bc64 with peer aad81047009a4ffcbc450b60030583b5
I20240501 08:12:48.842543 2134 ts_tablet_manager.cc:902] T d3673fe03fb64c47b30dc2b79042f44a P a29cd590999b4f32849ff5689cd53c11: Initiating tablet copy from peer 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:48.864663 2134 tablet_copy_client.cc:308] T d3673fe03fb64c47b30dc2b79042f44a P a29cd590999b4f32849ff5689cd53c11: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.135:33503
I20240501 08:12:48.865396 30825 tablet_copy_service.cc:140] P 64998323a1f4482d8704bf20594d13a5: Received BeginTabletCopySession request for tablet d3673fe03fb64c47b30dc2b79042f44a from peer a29cd590999b4f32849ff5689cd53c11 ({username='slave'} at 127.21.84.136:54515)
I20240501 08:12:48.865664 30825 tablet_copy_service.cc:161] P 64998323a1f4482d8704bf20594d13a5: Beginning new tablet copy session on tablet d3673fe03fb64c47b30dc2b79042f44a from peer a29cd590999b4f32849ff5689cd53c11 at {username='slave'} at 127.21.84.136:54515: session id = a29cd590999b4f32849ff5689cd53c11-d3673fe03fb64c47b30dc2b79042f44a
I20240501 08:12:48.868635 30825 tablet_copy_source_session.cc:205] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:48.870188 2134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3673fe03fb64c47b30dc2b79042f44a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:48.879565 2117 raft_consensus.cc:1050] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11: attempting to promote NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 to VOTER
I20240501 08:12:48.880477 2117 consensus_queue.cc:237] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:12:48.881691 2134 tablet_copy_client.cc:783] T d3673fe03fb64c47b30dc2b79042f44a P a29cd590999b4f32849ff5689cd53c11: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:48.882508 2134 tablet_copy_client.cc:647] T d3673fe03fb64c47b30dc2b79042f44a P a29cd590999b4f32849ff5689cd53c11: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:48.883409 30537 raft_consensus.cc:1261] T 8277d70d49f54197af60e0be86d34a9b P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:48.883668 30803 raft_consensus.cc:1261] T 8277d70d49f54197af60e0be86d34a9b P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:48.884390 2123 consensus_queue.cc:1035] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:48.886307 1940 consensus_queue.cc:1035] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:48.887984 30672 raft_consensus.cc:1261] T 8277d70d49f54197af60e0be86d34a9b P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:48.888860 1940 consensus_queue.cc:1035] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:48.884872 2134 tablet_copy_client.cc:523] T d3673fe03fb64c47b30dc2b79042f44a P a29cd590999b4f32849ff5689cd53c11: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:48.897598 2134 tablet_bootstrap.cc:492] T d3673fe03fb64c47b30dc2b79042f44a P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:48.908406 1940 raft_consensus.cc:2932] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:48.909539 30803 raft_consensus.cc:2932] T 8277d70d49f54197af60e0be86d34a9b P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:48.914671 30672 raft_consensus.cc:2932] T 8277d70d49f54197af60e0be86d34a9b P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:48.916565 30537 raft_consensus.cc:2932] T 8277d70d49f54197af60e0be86d34a9b P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:48.918152 27612 catalog_manager.cc:5484] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index 2 to 3, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:48.927515 2123 raft_consensus.cc:1050] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11: attempting to promote NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 to VOTER
I20240501 08:12:48.927595 30934 consensus_queue.cc:237] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:12:48.928378 2123 consensus_queue.cc:237] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:12:48.932058 30537 raft_consensus.cc:1261] T 8277d70d49f54197af60e0be86d34a9b P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:48.934255 2099 ts_tablet_manager.cc:902] T 0e06cf862c954e8fb1ca087ec5184497 P aad81047009a4ffcbc450b60030583b5: Initiating tablet copy from peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:12:48.935333 1378 consensus_queue.cc:1035] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:48.939652 30669 raft_consensus.cc:1261] T 8277d70d49f54197af60e0be86d34a9b P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:48.941823 2117 consensus_queue.cc:1035] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 08:12:48.943167 1378 raft_consensus.cc:2932] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:48.944216 28151 raft_consensus.cc:1261] T 82d6e5e399744b96a66373240ce1764d P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:48.943948 30669 raft_consensus.cc:2932] T 8277d70d49f54197af60e0be86d34a9b P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:48.945828 2117 consensus_queue.cc:1035] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:48.957654 2134 tablet_bootstrap.cc:492] T d3673fe03fb64c47b30dc2b79042f44a P a29cd590999b4f32849ff5689cd53c11: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:48.958226 27759 raft_consensus.cc:1261] T 82d6e5e399744b96a66373240ce1764d P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:48.958276 2134 tablet_bootstrap.cc:492] T d3673fe03fb64c47b30dc2b79042f44a P a29cd590999b4f32849ff5689cd53c11: Bootstrap complete.
I20240501 08:12:48.958639 2134 ts_tablet_manager.cc:1367] T d3673fe03fb64c47b30dc2b79042f44a P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.061s user 0.007s sys 0.005s
I20240501 08:12:48.959386 1378 consensus_queue.cc:1035] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:48.959796 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8277d70d49f54197af60e0be86d34a9b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:48.959512 2134 raft_consensus.cc:348] T d3673fe03fb64c47b30dc2b79042f44a P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:12:48.960325 2134 raft_consensus.cc:729] T d3673fe03fb64c47b30dc2b79042f44a P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: LEARNER
I20240501 08:12:48.960610 2134 consensus_queue.cc:260] T d3673fe03fb64c47b30dc2b79042f44a P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:12:48.961387 2134 ts_tablet_manager.cc:1398] T d3673fe03fb64c47b30dc2b79042f44a P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 08:12:48.961283 27620 catalog_manager.cc:5484] T 8277d70d49f54197af60e0be86d34a9b P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 3 to 4, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:48.942502 30537 raft_consensus.cc:2932] T 8277d70d49f54197af60e0be86d34a9b P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:48.965798 30784 tablet_service.cc:1514] Processing DeleteTablet for tablet 8277d70d49f54197af60e0be86d34a9b with delete_type TABLET_DATA_TOMBSTONED (TS 64998323a1f4482d8704bf20594d13a5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38952
I20240501 08:12:48.965801 30825 tablet_copy_service.cc:342] P 64998323a1f4482d8704bf20594d13a5: Request end of tablet copy session a29cd590999b4f32849ff5689cd53c11-d3673fe03fb64c47b30dc2b79042f44a received from {username='slave'} at 127.21.84.136:54515
I20240501 08:12:48.966445 30825 tablet_copy_service.cc:433] P 64998323a1f4482d8704bf20594d13a5: ending tablet copy session a29cd590999b4f32849ff5689cd53c11-d3673fe03fb64c47b30dc2b79042f44a on tablet d3673fe03fb64c47b30dc2b79042f44a with peer a29cd590999b4f32849ff5689cd53c11
I20240501 08:12:48.974417 30537 raft_consensus.cc:1261] T 82d6e5e399744b96a66373240ce1764d P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:48.974450 2099 tablet_copy_client.cc:308] T 0e06cf862c954e8fb1ca087ec5184497 P aad81047009a4ffcbc450b60030583b5: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.130:36503
I20240501 08:12:48.977031 27912 tablet_copy_service.cc:140] P f1525cbc82a24d13b65f136695c903af: Received BeginTabletCopySession request for tablet 0e06cf862c954e8fb1ca087ec5184497 from peer aad81047009a4ffcbc450b60030583b5 ({username='slave'} at 127.21.84.131:37997)
I20240501 08:12:48.977213 27912 tablet_copy_service.cc:161] P f1525cbc82a24d13b65f136695c903af: Beginning new tablet copy session on tablet 0e06cf862c954e8fb1ca087ec5184497 from peer aad81047009a4ffcbc450b60030583b5 at {username='slave'} at 127.21.84.131:37997: session id = aad81047009a4ffcbc450b60030583b5-0e06cf862c954e8fb1ca087ec5184497
I20240501 08:12:48.977332 1378 consensus_queue.cc:1035] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:48.979938 27912 tablet_copy_source_session.cc:205] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:48.981716 2099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e06cf862c954e8fb1ca087ec5184497. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:48.986578 2123 raft_consensus.cc:2932] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:48.992130 28151 raft_consensus.cc:2932] T 82d6e5e399744b96a66373240ce1764d P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:48.992588 2099 tablet_copy_client.cc:783] T 0e06cf862c954e8fb1ca087ec5184497 P aad81047009a4ffcbc450b60030583b5: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:48.992821 2099 tablet_copy_client.cc:647] T 0e06cf862c954e8fb1ca087ec5184497 P aad81047009a4ffcbc450b60030583b5: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:48.993978 27759 raft_consensus.cc:2932] T 82d6e5e399744b96a66373240ce1764d P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:48.998701 2099 tablet_copy_client.cc:523] T 0e06cf862c954e8fb1ca087ec5184497 P aad81047009a4ffcbc450b60030583b5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:49.002540 27612 catalog_manager.cc:5484] T 82d6e5e399744b96a66373240ce1764d P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:49.004398 2161 tablet_replica.cc:330] T 8277d70d49f54197af60e0be86d34a9b P 64998323a1f4482d8704bf20594d13a5: stopping tablet replica
I20240501 08:12:49.004760 2099 tablet_bootstrap.cc:492] T 0e06cf862c954e8fb1ca087ec5184497 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:49.004926 30537 raft_consensus.cc:2932] T 82d6e5e399744b96a66373240ce1764d P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:49.007481 2161 raft_consensus.cc:2229] T 8277d70d49f54197af60e0be86d34a9b P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:49.007824 2161 raft_consensus.cc:2258] T 8277d70d49f54197af60e0be86d34a9b P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:49.008786 2161 ts_tablet_manager.cc:1875] T 8277d70d49f54197af60e0be86d34a9b P 64998323a1f4482d8704bf20594d13a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:49.015775 30934 consensus_queue.cc:237] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:12:49.017727 28151 raft_consensus.cc:1261] T 82d6e5e399744b96a66373240ce1764d P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:49.018303 30534 raft_consensus.cc:1261] T 82d6e5e399744b96a66373240ce1764d P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:49.018944 1378 consensus_queue.cc:1035] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:49.019265 2123 consensus_queue.cc:1035] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:49.022519 1378 raft_consensus.cc:2932] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:49.030783 2161 ts_tablet_manager.cc:1888] T 8277d70d49f54197af60e0be86d34a9b P 64998323a1f4482d8704bf20594d13a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:49.031064 2161 log.cc:1196] T 8277d70d49f54197af60e0be86d34a9b P 64998323a1f4482d8704bf20594d13a5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal/wals/8277d70d49f54197af60e0be86d34a9b
I20240501 08:12:49.030953 30534 raft_consensus.cc:2932] T 82d6e5e399744b96a66373240ce1764d P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:49.031970 27605 catalog_manager.cc:4830] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): tablet 8277d70d49f54197af60e0be86d34a9b (table rebalancer_extra_table_14 [id=bd2ed215ec044ce782fe6f7c802797e6]) successfully deleted
I20240501 08:12:49.038185 27612 catalog_manager.cc:5484] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index 3 to 4, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:49.041538 28151 raft_consensus.cc:2932] T 82d6e5e399744b96a66373240ce1764d P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:49.047652 27741 tablet_service.cc:1514] Processing DeleteTablet for tablet 82d6e5e399744b96a66373240ce1764d with delete_type TABLET_DATA_TOMBSTONED (TS 6331f77dae9d452184e8cd0cbf5f1b43 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:49.048094 2101 tablet_replica.cc:330] T 82d6e5e399744b96a66373240ce1764d P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:49.048343 2101 raft_consensus.cc:2229] T 82d6e5e399744b96a66373240ce1764d P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:49.048573 2101 raft_consensus.cc:2258] T 82d6e5e399744b96a66373240ce1764d P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:49.049787 2099 tablet_bootstrap.cc:492] T 0e06cf862c954e8fb1ca087ec5184497 P aad81047009a4ffcbc450b60030583b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:49.049788 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 82d6e5e399744b96a66373240ce1764d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:49.050112 2101 ts_tablet_manager.cc:1875] T 82d6e5e399744b96a66373240ce1764d P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:49.051229 2099 tablet_bootstrap.cc:492] T 0e06cf862c954e8fb1ca087ec5184497 P aad81047009a4ffcbc450b60030583b5: Bootstrap complete.
I20240501 08:12:49.051482 2099 ts_tablet_manager.cc:1367] T 0e06cf862c954e8fb1ca087ec5184497 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.047s user 0.006s sys 0.003s
I20240501 08:12:49.052212 2099 raft_consensus.cc:348] T 0e06cf862c954e8fb1ca087ec5184497 P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:12:49.052500 2099 raft_consensus.cc:729] T 0e06cf862c954e8fb1ca087ec5184497 P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: LEARNER
I20240501 08:12:49.052695 2099 consensus_queue.cc:260] T 0e06cf862c954e8fb1ca087ec5184497 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:12:49.053491 2099 ts_tablet_manager.cc:1398] T 0e06cf862c954e8fb1ca087ec5184497 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:49.054275 27912 tablet_copy_service.cc:342] P f1525cbc82a24d13b65f136695c903af: Request end of tablet copy session aad81047009a4ffcbc450b60030583b5-0e06cf862c954e8fb1ca087ec5184497 received from {username='slave'} at 127.21.84.131:37997
I20240501 08:12:49.054391 27912 tablet_copy_service.cc:433] P f1525cbc82a24d13b65f136695c903af: ending tablet copy session aad81047009a4ffcbc450b60030583b5-0e06cf862c954e8fb1ca087ec5184497 on tablet 0e06cf862c954e8fb1ca087ec5184497 with peer aad81047009a4ffcbc450b60030583b5
I20240501 08:12:49.058923 2101 ts_tablet_manager.cc:1888] T 82d6e5e399744b96a66373240ce1764d P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:49.059139 2101 log.cc:1196] T 82d6e5e399744b96a66373240ce1764d P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/82d6e5e399744b96a66373240ce1764d
I20240501 08:12:49.060001 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet 82d6e5e399744b96a66373240ce1764d (table rebalancer_extra_table_32 [id=62ce67b20f6e4a2b80c594fcbe8f131e]) successfully deleted
I20240501 08:12:49.062255 28020 raft_consensus.cc:1203] T 7c3b2b31a9e747c69ff67969d878bc64 P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:49.108207 27889 raft_consensus.cc:1203] T 6c9bb7ab554e4023892339cd9e0c2a75 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:49.193303 27759 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f2baff8656804ce2bfb64a9280e16855"
dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48420
I20240501 08:12:49.193539 27759 raft_consensus.cc:595] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:12:49.222828 1740 raft_consensus.cc:1050] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6: attempting to promote NON_VOTER 31811708c6774037b82df44024b229b6 to VOTER
I20240501 08:12:49.223729 1740 consensus_queue.cc:237] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } }
I20240501 08:12:49.228137 27889 raft_consensus.cc:1261] T bc5778711a4c479881088af790f5b7d5 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:49.228792 1747 consensus_queue.cc:1035] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:49.230245 28016 raft_consensus.cc:1261] T bc5778711a4c479881088af790f5b7d5 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:49.232024 1747 consensus_queue.cc:1035] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:49.233258 30537 raft_consensus.cc:1261] T bc5778711a4c479881088af790f5b7d5 P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:49.235044 1735 consensus_queue.cc:1035] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:49.238525 1747 raft_consensus.cc:2932] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:49.250322 28016 raft_consensus.cc:2932] T bc5778711a4c479881088af790f5b7d5 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:49.255064 30537 raft_consensus.cc:2932] T bc5778711a4c479881088af790f5b7d5 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:49.259702 27612 catalog_manager.cc:5484] T bc5778711a4c479881088af790f5b7d5 P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 4 to 5, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:49.266654 30936 raft_consensus.cc:1203] T d3673fe03fb64c47b30dc2b79042f44a P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:49.276206 27889 raft_consensus.cc:2932] T bc5778711a4c479881088af790f5b7d5 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:49.284070 28152 consensus_queue.cc:237] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } }
I20240501 08:12:49.287820 28020 raft_consensus.cc:1261] T bc5778711a4c479881088af790f5b7d5 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:49.289000 30534 raft_consensus.cc:1261] T bc5778711a4c479881088af790f5b7d5 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:49.289877 1735 consensus_queue.cc:1035] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:49.291221 1747 consensus_queue.cc:1035] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:49.293782 1747 raft_consensus.cc:2932] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:49.296731 28022 raft_consensus.cc:2932] T bc5778711a4c479881088af790f5b7d5 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:49.301081 27759 raft_consensus.cc:1203] T 325b3dbba80d491dab81840e8eb171e9 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:49.301362 30534 raft_consensus.cc:2932] T bc5778711a4c479881088af790f5b7d5 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:49.308876 27604 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bc5778711a4c479881088af790f5b7d5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:49.322119 27612 catalog_manager.cc:5484] T bc5778711a4c479881088af790f5b7d5 P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index 5 to 6, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:12:49.328711 27871 tablet_service.cc:1514] Processing DeleteTablet for tablet bc5778711a4c479881088af790f5b7d5 with delete_type TABLET_DATA_TOMBSTONED (TS f1525cbc82a24d13b65f136695c903af not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:49.337893 2235 tablet_replica.cc:330] T bc5778711a4c479881088af790f5b7d5 P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:49.338229 2235 raft_consensus.cc:2229] T bc5778711a4c479881088af790f5b7d5 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:49.338510 2235 raft_consensus.cc:2258] T bc5778711a4c479881088af790f5b7d5 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:49.341404 2235 ts_tablet_manager.cc:1875] T bc5778711a4c479881088af790f5b7d5 P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 08:12:49.349768 27957 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.136:46845, user_credentials={real_user=slave}} blocked reactor thread for 51183.9us
I20240501 08:12:49.350332 2235 ts_tablet_manager.cc:1888] T bc5778711a4c479881088af790f5b7d5 P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:49.350557 2235 log.cc:1196] T bc5778711a4c479881088af790f5b7d5 P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/bc5778711a4c479881088af790f5b7d5
I20240501 08:12:49.351411 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet bc5778711a4c479881088af790f5b7d5 (table rebalance_test_table_5 [id=b19758166c02438c9fc4cfa835acc028]) successfully deleted
I20240501 08:12:49.365475 2198 raft_consensus.cc:979] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43: : Instructing follower aad81047009a4ffcbc450b60030583b5 to start an election
I20240501 08:12:49.365664 2198 raft_consensus.cc:1067] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Signalling peer aad81047009a4ffcbc450b60030583b5 to start an election
I20240501 08:12:49.369896 28022 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f2baff8656804ce2bfb64a9280e16855"
dest_uuid: "aad81047009a4ffcbc450b60030583b5"
from {username='slave'} at 127.21.84.129:39999
I20240501 08:12:49.370151 28022 raft_consensus.cc:482] T f2baff8656804ce2bfb64a9280e16855 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:49.370255 28022 raft_consensus.cc:3037] T f2baff8656804ce2bfb64a9280e16855 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:49.373018 28022 raft_consensus.cc:504] T f2baff8656804ce2bfb64a9280e16855 P aad81047009a4ffcbc450b60030583b5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:12:49.375118 28152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2baff8656804ce2bfb64a9280e16855" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:12:49.375614 28152 raft_consensus.cc:3037] T f2baff8656804ce2bfb64a9280e16855 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:49.376279 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2baff8656804ce2bfb64a9280e16855" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:12:49.376801 27759 raft_consensus.cc:3032] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:12:49.377084 27759 raft_consensus.cc:729] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:12:49.376989 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2baff8656804ce2bfb64a9280e16855" candidate_uuid: "aad81047009a4ffcbc450b60030583b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:49.377473 30672 raft_consensus.cc:3037] T f2baff8656804ce2bfb64a9280e16855 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:49.377714 27759 consensus_queue.cc:260] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:12:49.378187 27759 raft_consensus.cc:3037] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:49.380402 30672 raft_consensus.cc:2454] T f2baff8656804ce2bfb64a9280e16855 P 9a022e1a501f4d44a0468054d6c3de47 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 2.
I20240501 08:12:49.380506 28152 raft_consensus.cc:2454] T f2baff8656804ce2bfb64a9280e16855 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 2.
I20240501 08:12:49.381382 27955 leader_election.cc:302] T f2baff8656804ce2bfb64a9280e16855 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, 9a022e1a501f4d44a0468054d6c3de47, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:12:49.381685 1772 raft_consensus.cc:2781] T f2baff8656804ce2bfb64a9280e16855 P aad81047009a4ffcbc450b60030583b5 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:12:49.381804 1772 raft_consensus.cc:686] T f2baff8656804ce2bfb64a9280e16855 P aad81047009a4ffcbc450b60030583b5 [term 2 LEADER]: Becoming Leader. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Running, Role: LEADER
I20240501 08:12:49.382071 1772 consensus_queue.cc:237] T f2baff8656804ce2bfb64a9280e16855 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:12:49.384541 27759 raft_consensus.cc:2454] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad81047009a4ffcbc450b60030583b5 in term 2.
I20240501 08:12:49.389338 28022 leader_election.cc:288] T f2baff8656804ce2bfb64a9280e16855 P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 2 election: Requested vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:49.398840 27620 catalog_manager.cc:5484] T f2baff8656804ce2bfb64a9280e16855 P aad81047009a4ffcbc450b60030583b5 reported cstate change: term changed from 1 to 2, leader changed from 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) to aad81047009a4ffcbc450b60030583b5 (127.21.84.131). New cstate: current_term: 2 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:49.442050 30936 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b605254185d140d8ab90fcaa5229f168"
dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45574
I20240501 08:12:49.442281 30936 raft_consensus.cc:595] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:12:49.445389 28022 raft_consensus.cc:1203] T 0e06cf862c954e8fb1ca087ec5184497 P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:49.495656 1747 raft_consensus.cc:1050] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6: attempting to promote NON_VOTER aad81047009a4ffcbc450b60030583b5 to VOTER
I20240501 08:12:49.496534 1747 consensus_queue.cc:237] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } }
I20240501 08:12:49.497609 2062 raft_consensus.cc:1050] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5: attempting to promote NON_VOTER f1525cbc82a24d13b65f136695c903af to VOTER
I20240501 08:12:49.498598 2062 consensus_queue.cc:237] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:49.503365 30934 raft_consensus.cc:1261] T 7c3b2b31a9e747c69ff67969d878bc64 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:49.504015 2222 consensus_queue.cc:1035] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:49.504305 28022 raft_consensus.cc:1261] T 7c3b2b31a9e747c69ff67969d878bc64 P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:49.504335 30534 raft_consensus.cc:1261] T 7c3b2b31a9e747c69ff67969d878bc64 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:49.505059 2222 consensus_queue.cc:1035] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:49.505726 1740 consensus_queue.cc:1035] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:49.529326 27889 raft_consensus.cc:1261] T 6c9bb7ab554e4023892339cd9e0c2a75 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:49.529363 30934 raft_consensus.cc:1261] T 6c9bb7ab554e4023892339cd9e0c2a75 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:49.530102 2226 consensus_queue.cc:1035] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:49.529716 1740 raft_consensus.cc:2932] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:49.530594 2227 consensus_queue.cc:1035] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:49.530895 30536 raft_consensus.cc:2932] T 7c3b2b31a9e747c69ff67969d878bc64 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:49.532711 28022 raft_consensus.cc:2932] T 7c3b2b31a9e747c69ff67969d878bc64 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:49.539589 27620 catalog_manager.cc:5484] T 7c3b2b31a9e747c69ff67969d878bc64 P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:49.545029 30669 raft_consensus.cc:1261] T 6c9bb7ab554e4023892339cd9e0c2a75 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:49.549564 2226 consensus_queue.cc:1035] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:49.549662 30936 raft_consensus.cc:2932] T 7c3b2b31a9e747c69ff67969d878bc64 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:49.555011 28152 consensus_queue.cc:237] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } }
I20240501 08:12:49.560932 30536 raft_consensus.cc:1261] T 7c3b2b31a9e747c69ff67969d878bc64 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:49.562453 1747 consensus_queue.cc:1035] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:49.565858 1740 raft_consensus.cc:2932] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:49.568651 28022 raft_consensus.cc:1261] T 7c3b2b31a9e747c69ff67969d878bc64 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:49.571777 1735 consensus_queue.cc:1035] T 7c3b2b31a9e747c69ff67969d878bc64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:49.572954 30536 raft_consensus.cc:2932] T 7c3b2b31a9e747c69ff67969d878bc64 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:49.575877 27604 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7c3b2b31a9e747c69ff67969d878bc64 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:49.578176 28022 raft_consensus.cc:2932] T 7c3b2b31a9e747c69ff67969d878bc64 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:49.580637 27620 catalog_manager.cc:5484] T 7c3b2b31a9e747c69ff67969d878bc64 P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:49.582779 2117 raft_consensus.cc:979] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11: : Instructing follower f1525cbc82a24d13b65f136695c903af to start an election
I20240501 08:12:49.582919 2117 raft_consensus.cc:1067] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Signalling peer f1525cbc82a24d13b65f136695c903af to start an election
I20240501 08:12:49.583665 27891 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b605254185d140d8ab90fcaa5229f168"
dest_uuid: "f1525cbc82a24d13b65f136695c903af"
from {username='slave'} at 127.21.84.136:54747
I20240501 08:12:49.583850 27891 raft_consensus.cc:482] T b605254185d140d8ab90fcaa5229f168 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:49.583956 27891 raft_consensus.cc:3037] T b605254185d140d8ab90fcaa5229f168 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:49.584859 30916 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c3b2b31a9e747c69ff67969d878bc64 with delete_type TABLET_DATA_TOMBSTONED (TS a29cd590999b4f32849ff5689cd53c11 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54036
I20240501 08:12:49.585561 2264 tablet_replica.cc:330] T 7c3b2b31a9e747c69ff67969d878bc64 P a29cd590999b4f32849ff5689cd53c11: stopping tablet replica
I20240501 08:12:49.585863 2264 raft_consensus.cc:2229] T 7c3b2b31a9e747c69ff67969d878bc64 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:49.586184 2264 raft_consensus.cc:2258] T 7c3b2b31a9e747c69ff67969d878bc64 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:49.586656 27891 raft_consensus.cc:504] T b605254185d140d8ab90fcaa5229f168 P f1525cbc82a24d13b65f136695c903af [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:49.587463 27891 leader_election.cc:288] T b605254185d140d8ab90fcaa5229f168 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 2 election: Requested vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:49.588073 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b605254185d140d8ab90fcaa5229f168" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:49.588323 30936 raft_consensus.cc:3032] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:12:49.588397 30936 raft_consensus.cc:729] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:12:49.589259 30936 consensus_queue.cc:260] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:49.589735 30936 raft_consensus.cc:3037] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:49.589910 30672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b605254185d140d8ab90fcaa5229f168" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9a022e1a501f4d44a0468054d6c3de47"
I20240501 08:12:49.590235 30672 raft_consensus.cc:3037] T b605254185d140d8ab90fcaa5229f168 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:49.593482 28152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b605254185d140d8ab90fcaa5229f168" candidate_uuid: "f1525cbc82a24d13b65f136695c903af" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:12:49.593714 28152 raft_consensus.cc:3037] T b605254185d140d8ab90fcaa5229f168 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:49.593266 30672 raft_consensus.cc:2454] T b605254185d140d8ab90fcaa5229f168 P 9a022e1a501f4d44a0468054d6c3de47 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 2.
I20240501 08:12:49.597041 28152 raft_consensus.cc:2454] T b605254185d140d8ab90fcaa5229f168 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 2.
I20240501 08:12:49.597569 27825 leader_election.cc:302] T b605254185d140d8ab90fcaa5229f168 P f1525cbc82a24d13b65f136695c903af [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, 9a022e1a501f4d44a0468054d6c3de47, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:49.597977 2094 raft_consensus.cc:2781] T b605254185d140d8ab90fcaa5229f168 P f1525cbc82a24d13b65f136695c903af [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:12:49.598117 2094 raft_consensus.cc:686] T b605254185d140d8ab90fcaa5229f168 P f1525cbc82a24d13b65f136695c903af [term 2 LEADER]: Becoming Leader. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Running, Role: LEADER
I20240501 08:12:49.598425 2094 consensus_queue.cc:237] T b605254185d140d8ab90fcaa5229f168 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:49.600642 2227 raft_consensus.cc:2932] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:49.603282 30936 raft_consensus.cc:2454] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1525cbc82a24d13b65f136695c903af in term 2.
I20240501 08:12:49.604974 30936 raft_consensus.cc:2932] T 6c9bb7ab554e4023892339cd9e0c2a75 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:49.605523 30672 raft_consensus.cc:2932] T 6c9bb7ab554e4023892339cd9e0c2a75 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:49.605640 27891 raft_consensus.cc:2932] T 6c9bb7ab554e4023892339cd9e0c2a75 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:49.608376 27611 catalog_manager.cc:5484] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:49.613188 30800 consensus_queue.cc:237] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:49.615974 27889 raft_consensus.cc:1261] T 6c9bb7ab554e4023892339cd9e0c2a75 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:49.617081 1593 consensus_queue.cc:1035] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:49.617987 30934 raft_consensus.cc:1261] T 6c9bb7ab554e4023892339cd9e0c2a75 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:49.620009 2227 raft_consensus.cc:2932] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:49.620421 2226 consensus_queue.cc:1035] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:49.622892 27891 raft_consensus.cc:2932] T 6c9bb7ab554e4023892339cd9e0c2a75 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:49.624939 27605 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6c9bb7ab554e4023892339cd9e0c2a75 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:49.628360 27620 catalog_manager.cc:5484] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:49.633778 2264 ts_tablet_manager.cc:1875] T 7c3b2b31a9e747c69ff67969d878bc64 P a29cd590999b4f32849ff5689cd53c11: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:49.635401 30650 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c9bb7ab554e4023892339cd9e0c2a75 with delete_type TABLET_DATA_TOMBSTONED (TS 9a022e1a501f4d44a0468054d6c3de47 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57156
I20240501 08:12:49.635167 30934 raft_consensus.cc:2932] T 6c9bb7ab554e4023892339cd9e0c2a75 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:49.650381 2268 tablet_replica.cc:330] T 6c9bb7ab554e4023892339cd9e0c2a75 P 9a022e1a501f4d44a0468054d6c3de47: stopping tablet replica
I20240501 08:12:49.652014 2268 raft_consensus.cc:2229] T 6c9bb7ab554e4023892339cd9e0c2a75 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:49.651886 27620 catalog_manager.cc:5484] T b605254185d140d8ab90fcaa5229f168 P f1525cbc82a24d13b65f136695c903af reported cstate change: term changed from 1 to 2, leader changed from a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) to f1525cbc82a24d13b65f136695c903af (127.21.84.130). New cstate: current_term: 2 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:49.653334 2268 raft_consensus.cc:2258] T 6c9bb7ab554e4023892339cd9e0c2a75 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:49.654438 2268 ts_tablet_manager.cc:1875] T 6c9bb7ab554e4023892339cd9e0c2a75 P 9a022e1a501f4d44a0468054d6c3de47: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:49.662037 2264 ts_tablet_manager.cc:1888] T 7c3b2b31a9e747c69ff67969d878bc64 P a29cd590999b4f32849ff5689cd53c11: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:49.662246 2264 log.cc:1196] T 7c3b2b31a9e747c69ff67969d878bc64 P a29cd590999b4f32849ff5689cd53c11: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/wals/7c3b2b31a9e747c69ff67969d878bc64
I20240501 08:12:49.663619 27606 catalog_manager.cc:4830] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): tablet 7c3b2b31a9e747c69ff67969d878bc64 (table rebalancer_extra_table_18 [id=c906ec3019644e00bd7c63c8bcc3d323]) successfully deleted
I20240501 08:12:49.665683 2268 ts_tablet_manager.cc:1888] T 6c9bb7ab554e4023892339cd9e0c2a75 P 9a022e1a501f4d44a0468054d6c3de47: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:49.665804 2268 log.cc:1196] T 6c9bb7ab554e4023892339cd9e0c2a75 P 9a022e1a501f4d44a0468054d6c3de47: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/wals/6c9bb7ab554e4023892339cd9e0c2a75
I20240501 08:12:49.667263 27604 catalog_manager.cc:4830] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): tablet 6c9bb7ab554e4023892339cd9e0c2a75 (table rebalancer_extra_table_50 [id=95faa655da1346f68f96519e9154b964]) successfully deleted
I20240501 08:12:49.721258 1593 raft_consensus.cc:1050] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5: attempting to promote NON_VOTER a29cd590999b4f32849ff5689cd53c11 to VOTER
I20240501 08:12:49.724499 1593 consensus_queue.cc:237] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:12:49.732944 28152 raft_consensus.cc:1261] T d3673fe03fb64c47b30dc2b79042f44a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:49.733615 2216 consensus_queue.cc:1035] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:49.733808 30536 raft_consensus.cc:1261] T d3673fe03fb64c47b30dc2b79042f44a P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:49.734519 2216 consensus_queue.cc:1035] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:49.740514 30934 raft_consensus.cc:1261] T d3673fe03fb64c47b30dc2b79042f44a P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:49.741829 2062 consensus_queue.cc:1035] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20240501 08:12:49.757520 2216 raft_consensus.cc:2932] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:49.759338 30934 raft_consensus.cc:2932] T d3673fe03fb64c47b30dc2b79042f44a P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:49.762642 28152 raft_consensus.cc:2932] T d3673fe03fb64c47b30dc2b79042f44a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:49.762614 30536 raft_consensus.cc:2932] T d3673fe03fb64c47b30dc2b79042f44a P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:49.765218 30800 consensus_queue.cc:237] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:12:49.769968 27611 catalog_manager.cc:5484] T d3673fe03fb64c47b30dc2b79042f44a P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index 2 to 3, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:49.777735 30534 raft_consensus.cc:1261] T c5f4c78e69fb4c67ad65a7b0c1466798 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:49.778033 28020 raft_consensus.cc:1261] T c5f4c78e69fb4c67ad65a7b0c1466798 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:12:49.778033 30737 consensus_peers.cc:485] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 -> Peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Couldn't send request to peer 91d11eae5ef6432ca86eb8ed446b6ea6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5f4c78e69fb4c67ad65a7b0c1466798. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:49.779385 2229 consensus_queue.cc:1035] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:49.780511 2226 consensus_queue.cc:1035] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:49.782562 2227 raft_consensus.cc:2932] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:12:49.783164 30534 raft_consensus.cc:2932] T c5f4c78e69fb4c67ad65a7b0c1466798 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:12:49.787305 28020 raft_consensus.cc:2932] T c5f4c78e69fb4c67ad65a7b0c1466798 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:12:49.795013 27620 catalog_manager.cc:5484] T c5f4c78e69fb4c67ad65a7b0c1466798 P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:12:49.812327 28020 consensus_queue.cc:237] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:12:49.814581 30803 raft_consensus.cc:1261] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:49.815122 1772 consensus_queue.cc:1035] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:49.815184 30536 raft_consensus.cc:1261] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:49.815690 2232 consensus_queue.cc:1035] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:12:49.815975 27957 consensus_peers.cc:485] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 73f40a7ef6954b34a48d8c5aeeaebc0a. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:49.818922 2233 raft_consensus.cc:2932] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:49.819873 30534 raft_consensus.cc:2932] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:49.823697 30803 raft_consensus.cc:2932] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:49.830060 27620 catalog_manager.cc:5484] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:49.845273 27759 consensus_queue.cc:237] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
W20240501 08:12:49.847843 27694 consensus_peers.cc:485] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 -> Peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Couldn't send request to peer 9a022e1a501f4d44a0468054d6c3de47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e94895331d6b474484d987d42f13fd88. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:49.847849 30534 raft_consensus.cc:1261] T e94895331d6b474484d987d42f13fd88 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:49.848367 30803 raft_consensus.cc:1261] T e94895331d6b474484d987d42f13fd88 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:49.848381 2198 consensus_queue.cc:1035] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:49.848811 2198 consensus_queue.cc:1035] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:49.851361 1766 raft_consensus.cc:2932] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:49.852284 30534 raft_consensus.cc:2932] T e94895331d6b474484d987d42f13fd88 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:49.860194 27612 catalog_manager.cc:5484] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index -1 to 2, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:49.870008 30803 raft_consensus.cc:2932] T e94895331d6b474484d987d42f13fd88 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:49.871234 28152 raft_consensus.cc:1261] T f2baff8656804ce2bfb64a9280e16855 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:12:49.875653 1772 consensus_queue.cc:1035] T f2baff8656804ce2bfb64a9280e16855 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:49.885063 2227 raft_consensus.cc:1050] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5: attempting to promote NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 to VOTER
I20240501 08:12:49.885777 2227 consensus_queue.cc:237] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:12:49.888698 28151 raft_consensus.cc:1261] T 325b3dbba80d491dab81840e8eb171e9 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:49.889962 2229 consensus_queue.cc:1035] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:49.891088 30672 raft_consensus.cc:1261] T 325b3dbba80d491dab81840e8eb171e9 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:49.891806 2226 consensus_queue.cc:1035] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:49.894178 30669 raft_consensus.cc:1261] T f2baff8656804ce2bfb64a9280e16855 P 9a022e1a501f4d44a0468054d6c3de47 [term 2 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:12:49.895140 2231 consensus_queue.cc:1035] T f2baff8656804ce2bfb64a9280e16855 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:49.896684 27759 raft_consensus.cc:1261] T 325b3dbba80d491dab81840e8eb171e9 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:49.897464 2216 consensus_queue.cc:1035] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:49.919872 27758 raft_consensus.cc:1261] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:12:49.921895 2233 consensus_queue.cc:1035] T f2baff8656804ce2bfb64a9280e16855 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:49.939447 2229 raft_consensus.cc:2932] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:49.945775 30669 raft_consensus.cc:2932] T 325b3dbba80d491dab81840e8eb171e9 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:49.950706 28151 raft_consensus.cc:2932] T 325b3dbba80d491dab81840e8eb171e9 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:49.951824 27615 catalog_manager.cc:5484] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:49.960284 28020 consensus_queue.cc:237] T f2baff8656804ce2bfb64a9280e16855 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:12:49.972870 30803 consensus_queue.cc:237] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:12:49.976017 27760 raft_consensus.cc:1261] T 325b3dbba80d491dab81840e8eb171e9 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:49.976286 30672 raft_consensus.cc:1261] T f2baff8656804ce2bfb64a9280e16855 P 9a022e1a501f4d44a0468054d6c3de47 [term 2 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:12:49.976840 2216 consensus_queue.cc:1035] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:49.977289 1772 consensus_queue.cc:1035] T f2baff8656804ce2bfb64a9280e16855 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:49.978497 30670 raft_consensus.cc:1261] T 325b3dbba80d491dab81840e8eb171e9 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:49.979332 2226 consensus_queue.cc:1035] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:49.983734 1772 raft_consensus.cc:2932] T f2baff8656804ce2bfb64a9280e16855 P aad81047009a4ffcbc450b60030583b5 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:49.985320 2226 raft_consensus.cc:2932] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:49.987859 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f2baff8656804ce2bfb64a9280e16855 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:49.989297 27605 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 325b3dbba80d491dab81840e8eb171e9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:49.990537 27612 catalog_manager.cc:5484] T f2baff8656804ce2bfb64a9280e16855 P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 5 to 7, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New cstate: current_term: 2 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:49.990691 30669 raft_consensus.cc:2932] T f2baff8656804ce2bfb64a9280e16855 P 9a022e1a501f4d44a0468054d6c3de47 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:49.992144 30670 raft_consensus.cc:2932] T 325b3dbba80d491dab81840e8eb171e9 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:49.995249 27741 tablet_service.cc:1514] Processing DeleteTablet for tablet f2baff8656804ce2bfb64a9280e16855 with delete_type TABLET_DATA_TOMBSTONED (TS 6331f77dae9d452184e8cd0cbf5f1b43 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60422
I20240501 08:12:49.997768 27615 catalog_manager.cc:5484] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index 3 to 4, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:50.003953 28131 tablet_service.cc:1514] Processing DeleteTablet for tablet 325b3dbba80d491dab81840e8eb171e9 with delete_type TABLET_DATA_TOMBSTONED (TS 91d11eae5ef6432ca86eb8ed446b6ea6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:50.004582 2326 tablet_replica.cc:330] T 325b3dbba80d491dab81840e8eb171e9 P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:50.004846 2326 raft_consensus.cc:2229] T 325b3dbba80d491dab81840e8eb171e9 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:50.004964 28149 consensus_queue.cc:237] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
I20240501 08:12:50.005178 2326 raft_consensus.cc:2258] T 325b3dbba80d491dab81840e8eb171e9 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:50.006191 2326 ts_tablet_manager.cc:1875] T 325b3dbba80d491dab81840e8eb171e9 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:49.982463 27759 raft_consensus.cc:2932] T 325b3dbba80d491dab81840e8eb171e9 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:50.013173 2326 ts_tablet_manager.cc:1888] T 325b3dbba80d491dab81840e8eb171e9 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:50.013342 2326 log.cc:1196] T 325b3dbba80d491dab81840e8eb171e9 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/325b3dbba80d491dab81840e8eb171e9
I20240501 08:12:50.014348 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet 325b3dbba80d491dab81840e8eb171e9 (table rebalancer_extra_table_42 [id=6aa96b40a8e74b81bfc5ccab950a210d]) successfully deleted
I20240501 08:12:50.015615 2328 tablet_replica.cc:330] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:12:50.015898 2328 raft_consensus.cc:2229] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:50.016124 2328 pending_rounds.cc:62] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43: Trying to abort 1 pending ops.
I20240501 08:12:50.016232 2328 pending_rounds.cc:68] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43: Aborting op as it isn't in flight: id { term: 2 index: 6 } timestamp: 7022801589792440320 op_type: NO_OP noop_request { }
I20240501 08:12:50.016450 2328 raft_consensus.cc:2866] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240501 08:12:50.016615 2328 raft_consensus.cc:2258] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:50.017557 28151 raft_consensus.cc:1261] T f2baff8656804ce2bfb64a9280e16855 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:12:50.018465 2233 consensus_queue.cc:1035] T f2baff8656804ce2bfb64a9280e16855 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:50.019073 2328 ts_tablet_manager.cc:1875] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 08:12:50.019227 28086 consensus_peers.cc:485] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 -> Peer 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): Couldn't send request to peer 64998323a1f4482d8704bf20594d13a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a0f992f08fb841a5b18b14f5ad0b9c58. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:50.019243 28020 raft_consensus.cc:1261] T a0f992f08fb841a5b18b14f5ad0b9c58 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:50.019274 27891 raft_consensus.cc:1261] T a0f992f08fb841a5b18b14f5ad0b9c58 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:50.019806 1747 consensus_queue.cc:1035] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:50.020270 1747 consensus_queue.cc:1035] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:50.024950 28149 raft_consensus.cc:2932] T f2baff8656804ce2bfb64a9280e16855 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:50.028537 2328 ts_tablet_manager.cc:1888] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 08:12:50.028766 2328 log.cc:1196] T f2baff8656804ce2bfb64a9280e16855 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/f2baff8656804ce2bfb64a9280e16855
I20240501 08:12:50.029628 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet f2baff8656804ce2bfb64a9280e16855 (table rebalance_test_table_3 [id=56fa27c3d3de4793aadbcaf54cdb0ac5]) successfully deleted
I20240501 08:12:50.035838 27760 raft_consensus.cc:2932] T 325b3dbba80d491dab81840e8eb171e9 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:50.042326 2222 raft_consensus.cc:2932] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:12:50.042881 27889 raft_consensus.cc:2932] T a0f992f08fb841a5b18b14f5ad0b9c58 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:12:50.050695 28020 raft_consensus.cc:2932] T a0f992f08fb841a5b18b14f5ad0b9c58 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:12:50.053201 27620 catalog_manager.cc:5484] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: config changed from index -1 to 4, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:50.070072 2021 raft_consensus.cc:1050] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af: attempting to promote NON_VOTER aad81047009a4ffcbc450b60030583b5 to VOTER
I20240501 08:12:50.070928 2021 consensus_queue.cc:237] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } }
I20240501 08:12:50.079279 30803 raft_consensus.cc:1261] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:50.079891 2021 consensus_queue.cc:1035] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:50.080318 28020 raft_consensus.cc:1261] T 0e06cf862c954e8fb1ca087ec5184497 P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:50.080508 30534 raft_consensus.cc:1261] T 0e06cf862c954e8fb1ca087ec5184497 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:50.080936 2094 consensus_queue.cc:1035] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:50.081533 2094 consensus_queue.cc:1035] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:50.090957 28149 consensus_queue.cc:237] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:12:50.102347 2337 raft_consensus.cc:2932] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:50.104614 28016 raft_consensus.cc:2932] T 0e06cf862c954e8fb1ca087ec5184497 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
W20240501 08:12:50.105564 28087 consensus_peers.cc:485] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 -> Peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): Couldn't send request to peer a29cd590999b4f32849ff5689cd53c11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a05223cb2f20478b8d29ffa065b4c67b. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:50.104647 28022 raft_consensus.cc:1261] T a05223cb2f20478b8d29ffa065b4c67b P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:50.106205 27760 raft_consensus.cc:1261] T a05223cb2f20478b8d29ffa065b4c67b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:50.106714 1740 consensus_queue.cc:1035] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:50.107228 1740 consensus_queue.cc:1035] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:50.108258 30803 raft_consensus.cc:2932] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:50.123871 27615 catalog_manager.cc:5484] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:50.131258 30534 raft_consensus.cc:2932] T 0e06cf862c954e8fb1ca087ec5184497 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:50.136336 28152 raft_consensus.cc:1261] T b605254185d140d8ab90fcaa5229f168 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:12:50.136072 2222 raft_consensus.cc:2932] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:50.137514 2343 consensus_queue.cc:1035] T b605254185d140d8ab90fcaa5229f168 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 08:12:50.138617 27759 raft_consensus.cc:2932] T a05223cb2f20478b8d29ffa065b4c67b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:50.144896 27612 catalog_manager.cc:5484] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: config changed from index -1 to 4, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:50.146600 27891 consensus_queue.cc:237] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } }
I20240501 08:12:50.154917 28020 raft_consensus.cc:1261] T 0e06cf862c954e8fb1ca087ec5184497 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:50.156157 28022 raft_consensus.cc:2932] T a05223cb2f20478b8d29ffa065b4c67b P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:50.157874 2021 consensus_queue.cc:1035] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:50.159170 30803 raft_consensus.cc:1261] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:50.160758 2021 consensus_queue.cc:1035] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:50.164790 30672 raft_consensus.cc:1261] T b605254185d140d8ab90fcaa5229f168 P 9a022e1a501f4d44a0468054d6c3de47 [term 2 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:12:50.165557 2337 consensus_queue.cc:1035] T b605254185d140d8ab90fcaa5229f168 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:50.166622 2337 raft_consensus.cc:2932] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:50.168203 30803 raft_consensus.cc:2932] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:50.173671 28020 raft_consensus.cc:2932] T 0e06cf862c954e8fb1ca087ec5184497 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:50.174309 27607 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0e06cf862c954e8fb1ca087ec5184497 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:50.184391 27620 catalog_manager.cc:5484] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af reported cstate change: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:50.188937 30514 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e06cf862c954e8fb1ca087ec5184497 with delete_type TABLET_DATA_TOMBSTONED (TS 31811708c6774037b82df44024b229b6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45078
I20240501 08:12:50.190212 2352 tablet_replica.cc:330] T 0e06cf862c954e8fb1ca087ec5184497 P 31811708c6774037b82df44024b229b6: stopping tablet replica
I20240501 08:12:50.190681 2352 raft_consensus.cc:2229] T 0e06cf862c954e8fb1ca087ec5184497 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:50.191387 2352 raft_consensus.cc:2258] T 0e06cf862c954e8fb1ca087ec5184497 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:50.192708 2352 ts_tablet_manager.cc:1875] T 0e06cf862c954e8fb1ca087ec5184497 P 31811708c6774037b82df44024b229b6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:50.206809 30934 raft_consensus.cc:1261] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:12:50.207523 2094 consensus_queue.cc:1035] T b605254185d140d8ab90fcaa5229f168 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:50.216249 2352 ts_tablet_manager.cc:1888] T 0e06cf862c954e8fb1ca087ec5184497 P 31811708c6774037b82df44024b229b6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:50.216416 2352 log.cc:1196] T 0e06cf862c954e8fb1ca087ec5184497 P 31811708c6774037b82df44024b229b6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/wals/0e06cf862c954e8fb1ca087ec5184497
W20240501 08:12:50.222959 30470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.132:41323, user_credentials={real_user=slave}} blocked reactor thread for 65604.1us
W20240501 08:12:50.238668 30471 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.135:33503, user_credentials={real_user=slave}} blocked reactor thread for 60027.5us
I20240501 08:12:50.242046 27607 catalog_manager.cc:4830] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): tablet 0e06cf862c954e8fb1ca087ec5184497 (table rebalancer_extra_table_26 [id=444d193c9c4c4c16969e6c88e6e590c9]) successfully deleted
I20240501 08:12:50.255146 27891 consensus_queue.cc:237] T b605254185d140d8ab90fcaa5229f168 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:50.260280 28152 raft_consensus.cc:1261] T b605254185d140d8ab90fcaa5229f168 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:12:50.262692 30669 raft_consensus.cc:1261] T b605254185d140d8ab90fcaa5229f168 P 9a022e1a501f4d44a0468054d6c3de47 [term 2 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:12:50.263343 2337 consensus_queue.cc:1035] T b605254185d140d8ab90fcaa5229f168 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:50.263810 2337 consensus_queue.cc:1035] T b605254185d140d8ab90fcaa5229f168 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:50.266575 2343 raft_consensus.cc:2932] T b605254185d140d8ab90fcaa5229f168 P f1525cbc82a24d13b65f136695c903af [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:50.269484 28152 raft_consensus.cc:2932] T b605254185d140d8ab90fcaa5229f168 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:50.271843 27607 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b605254185d140d8ab90fcaa5229f168 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:50.280768 27612 catalog_manager.cc:5484] T b605254185d140d8ab90fcaa5229f168 P f1525cbc82a24d13b65f136695c903af reported cstate change: config changed from index 3 to 5, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New cstate: current_term: 2 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:50.283802 30669 raft_consensus.cc:2932] T b605254185d140d8ab90fcaa5229f168 P 9a022e1a501f4d44a0468054d6c3de47 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:50.288446 30916 tablet_service.cc:1514] Processing DeleteTablet for tablet b605254185d140d8ab90fcaa5229f168 with delete_type TABLET_DATA_TOMBSTONED (TS a29cd590999b4f32849ff5689cd53c11 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54036
I20240501 08:12:50.305476 2362 tablet_replica.cc:330] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11: stopping tablet replica
I20240501 08:12:50.305727 2362 raft_consensus.cc:2229] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:50.305548 30669 consensus_queue.cc:237] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:12:50.305967 2362 raft_consensus.cc:2258] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:50.307410 2362 ts_tablet_manager.cc:1875] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:50.311195 30536 raft_consensus.cc:1261] T 00a706ff4ea7456aacd2586791704e35 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:50.311442 28020 raft_consensus.cc:1261] T 00a706ff4ea7456aacd2586791704e35 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:12:50.311609 30607 consensus_peers.cc:485] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00a706ff4ea7456aacd2586791704e35. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:50.311786 1679 consensus_queue.cc:1035] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:50.312108 1976 consensus_queue.cc:1035] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:50.339848 1679 raft_consensus.cc:2932] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:50.350071 30536 raft_consensus.cc:2932] T 00a706ff4ea7456aacd2586791704e35 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:50.352999 2362 ts_tablet_manager.cc:1888] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240501 08:12:50.355581 27615 catalog_manager.cc:5484] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New cstate: current_term: 1 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:50.354918 2364 ts_tablet_manager.cc:902] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 6331f77dae9d452184e8cd0cbf5f1b43: Initiating tablet copy from peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:50.362485 2362 log.cc:1196] T b605254185d140d8ab90fcaa5229f168 P a29cd590999b4f32849ff5689cd53c11: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/wals/b605254185d140d8ab90fcaa5229f168
I20240501 08:12:50.373430 27606 catalog_manager.cc:4830] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): tablet b605254185d140d8ab90fcaa5229f168 (table rebalancer_extra_table_40 [id=b218cf6d6a9e4117abbfea2879ddcd54]) successfully deleted
I20240501 08:12:50.384814 28020 raft_consensus.cc:2932] T 00a706ff4ea7456aacd2586791704e35 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:50.425112 2376 ts_tablet_manager.cc:902] T c5f4c78e69fb4c67ad65a7b0c1466798 P 91d11eae5ef6432ca86eb8ed446b6ea6: Initiating tablet copy from peer 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:50.425915 2375 ts_tablet_manager.cc:902] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47: Initiating tablet copy from peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:50.438338 2364 tablet_copy_client.cc:308] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.131:36305
I20240501 08:12:50.438930 28042 tablet_copy_service.cc:140] P aad81047009a4ffcbc450b60030583b5: Received BeginTabletCopySession request for tablet 73f40a7ef6954b34a48d8c5aeeaebc0a from peer 6331f77dae9d452184e8cd0cbf5f1b43 ({username='slave'} at 127.21.84.129:39999)
I20240501 08:12:50.439072 28042 tablet_copy_service.cc:161] P aad81047009a4ffcbc450b60030583b5: Beginning new tablet copy session on tablet 73f40a7ef6954b34a48d8c5aeeaebc0a from peer 6331f77dae9d452184e8cd0cbf5f1b43 at {username='slave'} at 127.21.84.129:39999: session id = 6331f77dae9d452184e8cd0cbf5f1b43-73f40a7ef6954b34a48d8c5aeeaebc0a
I20240501 08:12:50.441738 30800 consensus_queue.cc:237] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:12:50.442315 28042 tablet_copy_source_session.cc:205] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:50.443710 2364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73f40a7ef6954b34a48d8c5aeeaebc0a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 08:12:50.450044 30737 consensus_peers.cc:485] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 -> Peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Couldn't send request to peer 91d11eae5ef6432ca86eb8ed446b6ea6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9df818d6cb4946989f8a4d2e2ce9b8bb. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:50.450194 30536 raft_consensus.cc:1261] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:50.450507 28020 raft_consensus.cc:1261] T 9df818d6cb4946989f8a4d2e2ce9b8bb P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:50.450764 2227 consensus_queue.cc:1035] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:50.453590 2229 consensus_queue.cc:1035] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:50.454191 2227 raft_consensus.cc:2932] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:12:50.455003 30536 raft_consensus.cc:2932] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:12:50.463362 27620 catalog_manager.cc:5484] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:50.472811 2375 tablet_copy_client.cc:308] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.129:43471
I20240501 08:12:50.473618 27781 tablet_copy_service.cc:140] P 6331f77dae9d452184e8cd0cbf5f1b43: Received BeginTabletCopySession request for tablet e94895331d6b474484d987d42f13fd88 from peer 9a022e1a501f4d44a0468054d6c3de47 ({username='slave'} at 127.21.84.134:48929)
I20240501 08:12:50.473773 27781 tablet_copy_service.cc:161] P 6331f77dae9d452184e8cd0cbf5f1b43: Beginning new tablet copy session on tablet e94895331d6b474484d987d42f13fd88 from peer 9a022e1a501f4d44a0468054d6c3de47 at {username='slave'} at 127.21.84.134:48929: session id = 9a022e1a501f4d44a0468054d6c3de47-e94895331d6b474484d987d42f13fd88
I20240501 08:12:50.475929 2364 tablet_copy_client.cc:783] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:50.476168 2364 tablet_copy_client.cc:647] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:50.476670 27781 tablet_copy_source_session.cc:205] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:50.479677 2375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e94895331d6b474484d987d42f13fd88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:50.480526 2364 tablet_copy_client.cc:523] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:50.486474 28020 raft_consensus.cc:2932] T 9df818d6cb4946989f8a4d2e2ce9b8bb P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:12:50.492084 2364 tablet_bootstrap.cc:492] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:50.512511 2375 tablet_copy_client.cc:783] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:50.512928 2375 tablet_copy_client.cc:647] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:50.516357 2376 tablet_copy_client.cc:308] T c5f4c78e69fb4c67ad65a7b0c1466798 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.135:33503
I20240501 08:12:50.517127 30825 tablet_copy_service.cc:140] P 64998323a1f4482d8704bf20594d13a5: Received BeginTabletCopySession request for tablet c5f4c78e69fb4c67ad65a7b0c1466798 from peer 91d11eae5ef6432ca86eb8ed446b6ea6 ({username='slave'} at 127.21.84.132:54591)
I20240501 08:12:50.517298 30825 tablet_copy_service.cc:161] P 64998323a1f4482d8704bf20594d13a5: Beginning new tablet copy session on tablet c5f4c78e69fb4c67ad65a7b0c1466798 from peer 91d11eae5ef6432ca86eb8ed446b6ea6 at {username='slave'} at 127.21.84.132:54591: session id = 91d11eae5ef6432ca86eb8ed446b6ea6-c5f4c78e69fb4c67ad65a7b0c1466798
I20240501 08:12:50.516896 2375 tablet_copy_client.cc:523] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:50.520344 30825 tablet_copy_source_session.cc:205] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:50.521680 2375 tablet_bootstrap.cc:492] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:50.521855 2376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5f4c78e69fb4c67ad65a7b0c1466798. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:50.524258 2364 tablet_bootstrap.cc:492] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:50.524699 2364 tablet_bootstrap.cc:492] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap complete.
I20240501 08:12:50.524953 2364 ts_tablet_manager.cc:1367] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.003s
I20240501 08:12:50.526062 2364 raft_consensus.cc:348] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:12:50.526553 2364 raft_consensus.cc:729] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: LEARNER
I20240501 08:12:50.529000 2376 tablet_copy_client.cc:783] T c5f4c78e69fb4c67ad65a7b0c1466798 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:50.529304 2376 tablet_copy_client.cc:647] T c5f4c78e69fb4c67ad65a7b0c1466798 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:50.529263 2364 consensus_queue.cc:260] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:12:50.531622 2376 tablet_copy_client.cc:523] T c5f4c78e69fb4c67ad65a7b0c1466798 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:50.535569 2376 tablet_bootstrap.cc:492] T c5f4c78e69fb4c67ad65a7b0c1466798 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:50.536854 2364 ts_tablet_manager.cc:1398] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.012s user 0.002s sys 0.002s
I20240501 08:12:50.538615 28042 tablet_copy_service.cc:342] P aad81047009a4ffcbc450b60030583b5: Request end of tablet copy session 6331f77dae9d452184e8cd0cbf5f1b43-73f40a7ef6954b34a48d8c5aeeaebc0a received from {username='slave'} at 127.21.84.129:39999
I20240501 08:12:50.538730 28042 tablet_copy_service.cc:433] P aad81047009a4ffcbc450b60030583b5: ending tablet copy session 6331f77dae9d452184e8cd0cbf5f1b43-73f40a7ef6954b34a48d8c5aeeaebc0a on tablet 73f40a7ef6954b34a48d8c5aeeaebc0a with peer 6331f77dae9d452184e8cd0cbf5f1b43
I20240501 08:12:50.555766 2375 tablet_bootstrap.cc:492] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:50.556833 2394 ts_tablet_manager.cc:902] T a0f992f08fb841a5b18b14f5ad0b9c58 P 64998323a1f4482d8704bf20594d13a5: Initiating tablet copy from peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:50.558917 2375 tablet_bootstrap.cc:492] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap complete.
I20240501 08:12:50.560688 2375 ts_tablet_manager.cc:1367] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.039s user 0.011s sys 0.000s
I20240501 08:12:50.560892 2391 ts_tablet_manager.cc:902] T a05223cb2f20478b8d29ffa065b4c67b P a29cd590999b4f32849ff5689cd53c11: Initiating tablet copy from peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:50.562000 2375 raft_consensus.cc:348] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:12:50.562326 2375 raft_consensus.cc:729] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: LEARNER
I20240501 08:12:50.562626 2375 consensus_queue.cc:260] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:12:50.575572 2375 ts_tablet_manager.cc:1398] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.015s user 0.003s sys 0.000s
I20240501 08:12:50.577592 27781 tablet_copy_service.cc:342] P 6331f77dae9d452184e8cd0cbf5f1b43: Request end of tablet copy session 9a022e1a501f4d44a0468054d6c3de47-e94895331d6b474484d987d42f13fd88 received from {username='slave'} at 127.21.84.134:48929
I20240501 08:12:50.577725 27781 tablet_copy_service.cc:433] P 6331f77dae9d452184e8cd0cbf5f1b43: ending tablet copy session 9a022e1a501f4d44a0468054d6c3de47-e94895331d6b474484d987d42f13fd88 on tablet e94895331d6b474484d987d42f13fd88 with peer 9a022e1a501f4d44a0468054d6c3de47
I20240501 08:12:50.582242 2376 tablet_bootstrap.cc:492] T c5f4c78e69fb4c67ad65a7b0c1466798 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:50.582674 2376 tablet_bootstrap.cc:492] T c5f4c78e69fb4c67ad65a7b0c1466798 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap complete.
I20240501 08:12:50.582927 2376 ts_tablet_manager.cc:1367] T c5f4c78e69fb4c67ad65a7b0c1466798 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.047s user 0.021s sys 0.017s
I20240501 08:12:50.583633 2376 raft_consensus.cc:348] T c5f4c78e69fb4c67ad65a7b0c1466798 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:12:50.584187 2376 raft_consensus.cc:729] T c5f4c78e69fb4c67ad65a7b0c1466798 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: LEARNER
I20240501 08:12:50.584396 2376 consensus_queue.cc:260] T c5f4c78e69fb4c67ad65a7b0c1466798 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:12:50.584918 2376 ts_tablet_manager.cc:1398] T c5f4c78e69fb4c67ad65a7b0c1466798 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:50.585582 30825 tablet_copy_service.cc:342] P 64998323a1f4482d8704bf20594d13a5: Request end of tablet copy session 91d11eae5ef6432ca86eb8ed446b6ea6-c5f4c78e69fb4c67ad65a7b0c1466798 received from {username='slave'} at 127.21.84.132:54591
I20240501 08:12:50.585729 30825 tablet_copy_service.cc:433] P 64998323a1f4482d8704bf20594d13a5: ending tablet copy session 91d11eae5ef6432ca86eb8ed446b6ea6-c5f4c78e69fb4c67ad65a7b0c1466798 on tablet c5f4c78e69fb4c67ad65a7b0c1466798 with peer 91d11eae5ef6432ca86eb8ed446b6ea6
I20240501 08:12:50.630095 30800 consensus_queue.cc:237] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:50.630393 2391 tablet_copy_client.cc:308] T a05223cb2f20478b8d29ffa065b4c67b P a29cd590999b4f32849ff5689cd53c11: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.132:41323
I20240501 08:12:50.630944 2394 tablet_copy_client.cc:308] T a0f992f08fb841a5b18b14f5ad0b9c58 P 64998323a1f4482d8704bf20594d13a5: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.132:41323
I20240501 08:12:50.631634 28172 tablet_copy_service.cc:140] P 91d11eae5ef6432ca86eb8ed446b6ea6: Received BeginTabletCopySession request for tablet a05223cb2f20478b8d29ffa065b4c67b from peer a29cd590999b4f32849ff5689cd53c11 ({username='slave'} at 127.21.84.136:39335)
I20240501 08:12:50.631773 28172 tablet_copy_service.cc:161] P 91d11eae5ef6432ca86eb8ed446b6ea6: Beginning new tablet copy session on tablet a05223cb2f20478b8d29ffa065b4c67b from peer a29cd590999b4f32849ff5689cd53c11 at {username='slave'} at 127.21.84.136:39335: session id = a29cd590999b4f32849ff5689cd53c11-a05223cb2f20478b8d29ffa065b4c67b
I20240501 08:12:50.631805 28171 tablet_copy_service.cc:140] P 91d11eae5ef6432ca86eb8ed446b6ea6: Received BeginTabletCopySession request for tablet a0f992f08fb841a5b18b14f5ad0b9c58 from peer 64998323a1f4482d8704bf20594d13a5 ({username='slave'} at 127.21.84.135:48917)
I20240501 08:12:50.631994 28171 tablet_copy_service.cc:161] P 91d11eae5ef6432ca86eb8ed446b6ea6: Beginning new tablet copy session on tablet a0f992f08fb841a5b18b14f5ad0b9c58 from peer 64998323a1f4482d8704bf20594d13a5 at {username='slave'} at 127.21.84.135:48917: session id = 64998323a1f4482d8704bf20594d13a5-a0f992f08fb841a5b18b14f5ad0b9c58
I20240501 08:12:50.634918 28172 tablet_copy_source_session.cc:205] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:50.634918 28171 tablet_copy_source_session.cc:205] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:50.636878 30672 raft_consensus.cc:1261] T f607d2e086cc4352a05a5ef2b02788cc P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:12:50.637429 30740 consensus_peers.cc:485] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f607d2e086cc4352a05a5ef2b02788cc. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:50.637713 2216 consensus_queue.cc:1035] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:50.638085 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0f992f08fb841a5b18b14f5ad0b9c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:50.637383 30934 raft_consensus.cc:1261] T f607d2e086cc4352a05a5ef2b02788cc P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:50.642992 2216 consensus_queue.cc:1035] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:50.636448 2391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a05223cb2f20478b8d29ffa065b4c67b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:50.652568 2227 raft_consensus.cc:2932] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:50.653304 30934 raft_consensus.cc:2932] T f607d2e086cc4352a05a5ef2b02788cc P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:50.664917 27620 catalog_manager.cc:5484] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:50.666999 2391 tablet_copy_client.cc:783] T a05223cb2f20478b8d29ffa065b4c67b P a29cd590999b4f32849ff5689cd53c11: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:50.667289 2391 tablet_copy_client.cc:647] T a05223cb2f20478b8d29ffa065b4c67b P a29cd590999b4f32849ff5689cd53c11: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:50.668031 2394 tablet_copy_client.cc:783] T a0f992f08fb841a5b18b14f5ad0b9c58 P 64998323a1f4482d8704bf20594d13a5: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:50.669198 30670 raft_consensus.cc:2932] T f607d2e086cc4352a05a5ef2b02788cc P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:50.680456 2394 tablet_copy_client.cc:647] T a0f992f08fb841a5b18b14f5ad0b9c58 P 64998323a1f4482d8704bf20594d13a5: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:50.682575 2391 tablet_copy_client.cc:523] T a05223cb2f20478b8d29ffa065b4c67b P a29cd590999b4f32849ff5689cd53c11: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:50.684301 2394 tablet_copy_client.cc:523] T a0f992f08fb841a5b18b14f5ad0b9c58 P 64998323a1f4482d8704bf20594d13a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:50.692589 2391 tablet_bootstrap.cc:492] T a05223cb2f20478b8d29ffa065b4c67b P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:50.694298 2394 tablet_bootstrap.cc:492] T a0f992f08fb841a5b18b14f5ad0b9c58 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:50.736250 30536 consensus_queue.cc:237] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
W20240501 08:12:50.745607 30472 consensus_peers.cc:485] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 -> Peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Couldn't send request to peer aad81047009a4ffcbc450b60030583b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b16391636d5f4ec48c9e4f18ed40badd. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:50.745869 30800 raft_consensus.cc:1261] T b16391636d5f4ec48c9e4f18ed40badd P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:50.746562 2353 consensus_queue.cc:1035] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:50.756191 27759 raft_consensus.cc:1261] T b16391636d5f4ec48c9e4f18ed40badd P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:50.756969 2353 consensus_queue.cc:1035] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:50.760514 2353 raft_consensus.cc:2932] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:50.761023 2391 tablet_bootstrap.cc:492] T a05223cb2f20478b8d29ffa065b4c67b P a29cd590999b4f32849ff5689cd53c11: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:50.761540 2391 tablet_bootstrap.cc:492] T a05223cb2f20478b8d29ffa065b4c67b P a29cd590999b4f32849ff5689cd53c11: Bootstrap complete.
I20240501 08:12:50.761850 2391 ts_tablet_manager.cc:1367] T a05223cb2f20478b8d29ffa065b4c67b P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.069s user 0.046s sys 0.009s
I20240501 08:12:50.763370 2391 raft_consensus.cc:348] T a05223cb2f20478b8d29ffa065b4c67b P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:12:50.763618 2391 raft_consensus.cc:729] T a05223cb2f20478b8d29ffa065b4c67b P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: LEARNER
I20240501 08:12:50.763818 2391 consensus_queue.cc:260] T a05223cb2f20478b8d29ffa065b4c67b P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:12:50.764254 30800 raft_consensus.cc:2932] T b16391636d5f4ec48c9e4f18ed40badd P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:50.768831 2391 ts_tablet_manager.cc:1398] T a05223cb2f20478b8d29ffa065b4c67b P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 08:12:50.773725 28171 tablet_copy_service.cc:342] P 91d11eae5ef6432ca86eb8ed446b6ea6: Request end of tablet copy session a29cd590999b4f32849ff5689cd53c11-a05223cb2f20478b8d29ffa065b4c67b received from {username='slave'} at 127.21.84.136:39335
I20240501 08:12:50.773870 28171 tablet_copy_service.cc:433] P 91d11eae5ef6432ca86eb8ed446b6ea6: ending tablet copy session a29cd590999b4f32849ff5689cd53c11-a05223cb2f20478b8d29ffa065b4c67b on tablet a05223cb2f20478b8d29ffa065b4c67b with peer a29cd590999b4f32849ff5689cd53c11
I20240501 08:12:50.776064 27612 catalog_manager.cc:5484] T b16391636d5f4ec48c9e4f18ed40badd P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:50.780360 2394 tablet_bootstrap.cc:492] T a0f992f08fb841a5b18b14f5ad0b9c58 P 64998323a1f4482d8704bf20594d13a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:50.781308 2394 tablet_bootstrap.cc:492] T a0f992f08fb841a5b18b14f5ad0b9c58 P 64998323a1f4482d8704bf20594d13a5: Bootstrap complete.
I20240501 08:12:50.781628 2394 ts_tablet_manager.cc:1367] T a0f992f08fb841a5b18b14f5ad0b9c58 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.087s user 0.040s sys 0.017s
I20240501 08:12:50.782438 2394 raft_consensus.cc:348] T a0f992f08fb841a5b18b14f5ad0b9c58 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
I20240501 08:12:50.782697 2394 raft_consensus.cc:729] T a0f992f08fb841a5b18b14f5ad0b9c58 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: LEARNER
I20240501 08:12:50.782935 2394 consensus_queue.cc:260] T a0f992f08fb841a5b18b14f5ad0b9c58 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
I20240501 08:12:50.798271 27759 raft_consensus.cc:2932] T b16391636d5f4ec48c9e4f18ed40badd P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:50.798676 2394 ts_tablet_manager.cc:1398] T a0f992f08fb841a5b18b14f5ad0b9c58 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:50.799753 28171 tablet_copy_service.cc:342] P 91d11eae5ef6432ca86eb8ed446b6ea6: Request end of tablet copy session 64998323a1f4482d8704bf20594d13a5-a0f992f08fb841a5b18b14f5ad0b9c58 received from {username='slave'} at 127.21.84.135:48917
I20240501 08:12:50.799881 28171 tablet_copy_service.cc:433] P 91d11eae5ef6432ca86eb8ed446b6ea6: ending tablet copy session 64998323a1f4482d8704bf20594d13a5-a0f992f08fb841a5b18b14f5ad0b9c58 on tablet a0f992f08fb841a5b18b14f5ad0b9c58 with peer 64998323a1f4482d8704bf20594d13a5
I20240501 08:12:50.807363 2364 ts_tablet_manager.cc:902] T 00a706ff4ea7456aacd2586791704e35 P 6331f77dae9d452184e8cd0cbf5f1b43: Initiating tablet copy from peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:50.812155 2364 tablet_copy_client.cc:308] T 00a706ff4ea7456aacd2586791704e35 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.134:39933
I20240501 08:12:50.812810 30693 tablet_copy_service.cc:140] P 9a022e1a501f4d44a0468054d6c3de47: Received BeginTabletCopySession request for tablet 00a706ff4ea7456aacd2586791704e35 from peer 6331f77dae9d452184e8cd0cbf5f1b43 ({username='slave'} at 127.21.84.129:59569)
I20240501 08:12:50.813014 30693 tablet_copy_service.cc:161] P 9a022e1a501f4d44a0468054d6c3de47: Beginning new tablet copy session on tablet 00a706ff4ea7456aacd2586791704e35 from peer 6331f77dae9d452184e8cd0cbf5f1b43 at {username='slave'} at 127.21.84.129:59569: session id = 6331f77dae9d452184e8cd0cbf5f1b43-00a706ff4ea7456aacd2586791704e35
I20240501 08:12:50.814941 30693 tablet_copy_source_session.cc:205] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:50.816231 2364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00a706ff4ea7456aacd2586791704e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:50.826522 2364 tablet_copy_client.cc:783] T 00a706ff4ea7456aacd2586791704e35 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:50.827337 2364 tablet_copy_client.cc:647] T 00a706ff4ea7456aacd2586791704e35 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:50.829792 2364 tablet_copy_client.cc:523] T 00a706ff4ea7456aacd2586791704e35 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:50.832659 2364 tablet_bootstrap.cc:492] T 00a706ff4ea7456aacd2586791704e35 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:50.849361 2364 tablet_bootstrap.cc:492] T 00a706ff4ea7456aacd2586791704e35 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:50.849948 2364 tablet_bootstrap.cc:492] T 00a706ff4ea7456aacd2586791704e35 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap complete.
I20240501 08:12:50.850322 2364 ts_tablet_manager.cc:1367] T 00a706ff4ea7456aacd2586791704e35 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.008s
I20240501 08:12:50.854753 2364 raft_consensus.cc:348] T 00a706ff4ea7456aacd2586791704e35 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:12:50.855127 2364 raft_consensus.cc:729] T 00a706ff4ea7456aacd2586791704e35 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: LEARNER
I20240501 08:12:50.855443 2364 consensus_queue.cc:260] T 00a706ff4ea7456aacd2586791704e35 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:12:50.861604 2364 ts_tablet_manager.cc:1398] T 00a706ff4ea7456aacd2586791704e35 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.011s user 0.002s sys 0.001s
I20240501 08:12:50.862455 30693 tablet_copy_service.cc:342] P 9a022e1a501f4d44a0468054d6c3de47: Request end of tablet copy session 6331f77dae9d452184e8cd0cbf5f1b43-00a706ff4ea7456aacd2586791704e35 received from {username='slave'} at 127.21.84.129:59569
I20240501 08:12:50.862607 30693 tablet_copy_service.cc:433] P 9a022e1a501f4d44a0468054d6c3de47: ending tablet copy session 6331f77dae9d452184e8cd0cbf5f1b43-00a706ff4ea7456aacd2586791704e35 on tablet 00a706ff4ea7456aacd2586791704e35 with peer 6331f77dae9d452184e8cd0cbf5f1b43
I20240501 08:12:50.873798 30670 consensus_queue.cc:237] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:12:50.876786 30934 raft_consensus.cc:1261] T 28a1f9ac749143de951c1b41beec9875 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:50.876888 30800 raft_consensus.cc:1261] T 28a1f9ac749143de951c1b41beec9875 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:50.877420 2436 consensus_queue.cc:1035] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:12:50.877816 30607 consensus_peers.cc:485] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 28a1f9ac749143de951c1b41beec9875. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:50.877907 1679 consensus_queue.cc:1035] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:50.880540 2436 raft_consensus.cc:2932] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:50.881171 30934 raft_consensus.cc:2932] T 28a1f9ac749143de951c1b41beec9875 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:50.891759 30670 raft_consensus.cc:1203] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:50.897267 27620 catalog_manager.cc:5484] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New cstate: current_term: 1 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:50.901793 30800 raft_consensus.cc:2932] T 28a1f9ac749143de951c1b41beec9875 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:50.929123 27759 raft_consensus.cc:1203] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:50.958043 28152 raft_consensus.cc:1203] T c5f4c78e69fb4c67ad65a7b0c1466798 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:50.958957 2376 ts_tablet_manager.cc:902] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 91d11eae5ef6432ca86eb8ed446b6ea6: Initiating tablet copy from peer 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:50.990710 2376 tablet_copy_client.cc:308] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.135:33503
I20240501 08:12:50.991623 30825 tablet_copy_service.cc:140] P 64998323a1f4482d8704bf20594d13a5: Received BeginTabletCopySession request for tablet 9df818d6cb4946989f8a4d2e2ce9b8bb from peer 91d11eae5ef6432ca86eb8ed446b6ea6 ({username='slave'} at 127.21.84.132:54591)
I20240501 08:12:50.991883 30825 tablet_copy_service.cc:161] P 64998323a1f4482d8704bf20594d13a5: Beginning new tablet copy session on tablet 9df818d6cb4946989f8a4d2e2ce9b8bb from peer 91d11eae5ef6432ca86eb8ed446b6ea6 at {username='slave'} at 127.21.84.132:54591: session id = 91d11eae5ef6432ca86eb8ed446b6ea6-9df818d6cb4946989f8a4d2e2ce9b8bb
I20240501 08:12:50.995527 30825 tablet_copy_source_session.cc:205] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:50.996137 30800 raft_consensus.cc:1203] T a0f992f08fb841a5b18b14f5ad0b9c58 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:51.003317 2376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9df818d6cb4946989f8a4d2e2ce9b8bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:51.013384 2376 tablet_copy_client.cc:783] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:51.013700 2376 tablet_copy_client.cc:647] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:51.015524 2376 tablet_copy_client.cc:523] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:51.019980 2376 tablet_bootstrap.cc:492] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:51.032618 2376 tablet_bootstrap.cc:492] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:51.033035 2376 tablet_bootstrap.cc:492] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap complete.
I20240501 08:12:51.033321 2376 ts_tablet_manager.cc:1367] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.006s
I20240501 08:12:51.034233 2376 raft_consensus.cc:348] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:12:51.034564 2376 raft_consensus.cc:729] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: LEARNER
I20240501 08:12:51.034786 2376 consensus_queue.cc:260] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:12:51.035521 2376 ts_tablet_manager.cc:1398] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:12:51.038667 2466 ts_tablet_manager.cc:902] T f607d2e086cc4352a05a5ef2b02788cc P f1525cbc82a24d13b65f136695c903af: Initiating tablet copy from peer 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:51.041296 30825 tablet_copy_service.cc:342] P 64998323a1f4482d8704bf20594d13a5: Request end of tablet copy session 91d11eae5ef6432ca86eb8ed446b6ea6-9df818d6cb4946989f8a4d2e2ce9b8bb received from {username='slave'} at 127.21.84.132:54591
I20240501 08:12:51.041440 30825 tablet_copy_service.cc:433] P 64998323a1f4482d8704bf20594d13a5: ending tablet copy session 91d11eae5ef6432ca86eb8ed446b6ea6-9df818d6cb4946989f8a4d2e2ce9b8bb on tablet 9df818d6cb4946989f8a4d2e2ce9b8bb with peer 91d11eae5ef6432ca86eb8ed446b6ea6
I20240501 08:12:51.041429 2466 tablet_copy_client.cc:308] T f607d2e086cc4352a05a5ef2b02788cc P f1525cbc82a24d13b65f136695c903af: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.135:33503
I20240501 08:12:51.042520 30825 tablet_copy_service.cc:140] P 64998323a1f4482d8704bf20594d13a5: Received BeginTabletCopySession request for tablet f607d2e086cc4352a05a5ef2b02788cc from peer f1525cbc82a24d13b65f136695c903af ({username='slave'} at 127.21.84.130:45079)
I20240501 08:12:51.042681 30825 tablet_copy_service.cc:161] P 64998323a1f4482d8704bf20594d13a5: Beginning new tablet copy session on tablet f607d2e086cc4352a05a5ef2b02788cc from peer f1525cbc82a24d13b65f136695c903af at {username='slave'} at 127.21.84.130:45079: session id = f1525cbc82a24d13b65f136695c903af-f607d2e086cc4352a05a5ef2b02788cc
I20240501 08:12:51.045522 30825 tablet_copy_source_session.cc:205] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:51.047248 2466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f607d2e086cc4352a05a5ef2b02788cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:51.053784 2466 tablet_copy_client.cc:783] T f607d2e086cc4352a05a5ef2b02788cc P f1525cbc82a24d13b65f136695c903af: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:51.054068 2466 tablet_copy_client.cc:647] T f607d2e086cc4352a05a5ef2b02788cc P f1525cbc82a24d13b65f136695c903af: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:51.067751 2466 tablet_copy_client.cc:523] T f607d2e086cc4352a05a5ef2b02788cc P f1525cbc82a24d13b65f136695c903af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:51.072732 2466 tablet_bootstrap.cc:492] T f607d2e086cc4352a05a5ef2b02788cc P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:51.088426 30934 raft_consensus.cc:1203] T a05223cb2f20478b8d29ffa065b4c67b P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:51.099753 2466 tablet_bootstrap.cc:492] T f607d2e086cc4352a05a5ef2b02788cc P f1525cbc82a24d13b65f136695c903af: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:51.100118 2466 tablet_bootstrap.cc:492] T f607d2e086cc4352a05a5ef2b02788cc P f1525cbc82a24d13b65f136695c903af: Bootstrap complete.
I20240501 08:12:51.100323 2466 ts_tablet_manager.cc:1367] T f607d2e086cc4352a05a5ef2b02788cc P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.006s
I20240501 08:12:51.100816 2466 raft_consensus.cc:348] T f607d2e086cc4352a05a5ef2b02788cc P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:51.101011 2466 raft_consensus.cc:729] T f607d2e086cc4352a05a5ef2b02788cc P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: LEARNER
I20240501 08:12:51.101215 2466 consensus_queue.cc:260] T f607d2e086cc4352a05a5ef2b02788cc P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:51.101650 2466 ts_tablet_manager.cc:1398] T f607d2e086cc4352a05a5ef2b02788cc P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:12:51.103896 30825 tablet_copy_service.cc:342] P 64998323a1f4482d8704bf20594d13a5: Request end of tablet copy session f1525cbc82a24d13b65f136695c903af-f607d2e086cc4352a05a5ef2b02788cc received from {username='slave'} at 127.21.84.130:45079
I20240501 08:12:51.104063 30825 tablet_copy_service.cc:433] P 64998323a1f4482d8704bf20594d13a5: ending tablet copy session f1525cbc82a24d13b65f136695c903af-f607d2e086cc4352a05a5ef2b02788cc on tablet f607d2e086cc4352a05a5ef2b02788cc with peer f1525cbc82a24d13b65f136695c903af
I20240501 08:12:51.145264 30800 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d3673fe03fb64c47b30dc2b79042f44a"
dest_uuid: "64998323a1f4482d8704bf20594d13a5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44962
I20240501 08:12:51.145591 30800 raft_consensus.cc:595] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:12:51.170703 2227 raft_consensus.cc:979] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5: : Instructing follower a29cd590999b4f32849ff5689cd53c11 to start an election
I20240501 08:12:51.170907 2227 raft_consensus.cc:1067] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Signalling peer a29cd590999b4f32849ff5689cd53c11 to start an election
I20240501 08:12:51.171866 30934 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3673fe03fb64c47b30dc2b79042f44a"
dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
from {username='slave'} at 127.21.84.135:35843
I20240501 08:12:51.172075 30934 raft_consensus.cc:482] T d3673fe03fb64c47b30dc2b79042f44a P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:51.172216 30934 raft_consensus.cc:3037] T d3673fe03fb64c47b30dc2b79042f44a P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:51.177431 30934 raft_consensus.cc:504] T d3673fe03fb64c47b30dc2b79042f44a P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:12:51.178251 30934 leader_election.cc:288] T d3673fe03fb64c47b30dc2b79042f44a P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 election: Requested vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:51.178442 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3673fe03fb64c47b30dc2b79042f44a" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:51.178675 30803 raft_consensus.cc:3032] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:12:51.178630 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3673fe03fb64c47b30dc2b79042f44a" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:51.178795 30803 raft_consensus.cc:729] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Running, Role: LEADER
I20240501 08:12:51.178790 28152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3673fe03fb64c47b30dc2b79042f44a" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:12:51.178896 30536 raft_consensus.cc:3037] T d3673fe03fb64c47b30dc2b79042f44a P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:51.178972 28152 raft_consensus.cc:3037] T d3673fe03fb64c47b30dc2b79042f44a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:51.179106 30803 consensus_queue.cc:260] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:12:51.179548 30803 raft_consensus.cc:3037] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:51.181293 30536 raft_consensus.cc:2454] T d3673fe03fb64c47b30dc2b79042f44a P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 2.
I20240501 08:12:51.181680 28152 raft_consensus.cc:2454] T d3673fe03fb64c47b30dc2b79042f44a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 2.
I20240501 08:12:51.182703 30803 raft_consensus.cc:2454] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 2.
I20240501 08:12:51.183178 30871 leader_election.cc:302] T d3673fe03fb64c47b30dc2b79042f44a P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 64998323a1f4482d8704bf20594d13a5, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:12:51.183465 2123 raft_consensus.cc:2781] T d3673fe03fb64c47b30dc2b79042f44a P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:12:51.183634 2123 raft_consensus.cc:686] T d3673fe03fb64c47b30dc2b79042f44a P a29cd590999b4f32849ff5689cd53c11 [term 2 LEADER]: Becoming Leader. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:12:51.183905 2123 consensus_queue.cc:237] T d3673fe03fb64c47b30dc2b79042f44a P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:12:51.189606 27620 catalog_manager.cc:5484] T d3673fe03fb64c47b30dc2b79042f44a P a29cd590999b4f32849ff5689cd53c11 reported cstate change: term changed from 1 to 2, leader changed from 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) to a29cd590999b4f32849ff5689cd53c11 (127.21.84.136). New cstate: current_term: 2 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:51.290716 2548 ts_tablet_manager.cc:902] T b16391636d5f4ec48c9e4f18ed40badd P aad81047009a4ffcbc450b60030583b5: Initiating tablet copy from peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:51.296713 27760 raft_consensus.cc:1203] T 00a706ff4ea7456aacd2586791704e35 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:51.304564 2548 tablet_copy_client.cc:308] T b16391636d5f4ec48c9e4f18ed40badd P aad81047009a4ffcbc450b60030583b5: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.133:38293
I20240501 08:12:51.305374 30559 tablet_copy_service.cc:140] P 31811708c6774037b82df44024b229b6: Received BeginTabletCopySession request for tablet b16391636d5f4ec48c9e4f18ed40badd from peer aad81047009a4ffcbc450b60030583b5 ({username='slave'} at 127.21.84.131:56895)
I20240501 08:12:51.305620 30559 tablet_copy_service.cc:161] P 31811708c6774037b82df44024b229b6: Beginning new tablet copy session on tablet b16391636d5f4ec48c9e4f18ed40badd from peer aad81047009a4ffcbc450b60030583b5 at {username='slave'} at 127.21.84.131:56895: session id = aad81047009a4ffcbc450b60030583b5-b16391636d5f4ec48c9e4f18ed40badd
I20240501 08:12:51.308782 30559 tablet_copy_source_session.cc:205] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:51.311012 2548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b16391636d5f4ec48c9e4f18ed40badd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:51.318115 2548 tablet_copy_client.cc:783] T b16391636d5f4ec48c9e4f18ed40badd P aad81047009a4ffcbc450b60030583b5: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:51.318361 2548 tablet_copy_client.cc:647] T b16391636d5f4ec48c9e4f18ed40badd P aad81047009a4ffcbc450b60030583b5: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:51.320046 2548 tablet_copy_client.cc:523] T b16391636d5f4ec48c9e4f18ed40badd P aad81047009a4ffcbc450b60030583b5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:51.323838 2548 tablet_bootstrap.cc:492] T b16391636d5f4ec48c9e4f18ed40badd P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:51.338858 2548 tablet_bootstrap.cc:492] T b16391636d5f4ec48c9e4f18ed40badd P aad81047009a4ffcbc450b60030583b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:51.339284 2548 tablet_bootstrap.cc:492] T b16391636d5f4ec48c9e4f18ed40badd P aad81047009a4ffcbc450b60030583b5: Bootstrap complete.
I20240501 08:12:51.339609 2548 ts_tablet_manager.cc:1367] T b16391636d5f4ec48c9e4f18ed40badd P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20240501 08:12:51.341054 2548 raft_consensus.cc:348] T b16391636d5f4ec48c9e4f18ed40badd P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:12:51.341570 2548 raft_consensus.cc:729] T b16391636d5f4ec48c9e4f18ed40badd P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: LEARNER
I20240501 08:12:51.342252 2548 consensus_queue.cc:260] T b16391636d5f4ec48c9e4f18ed40badd P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:12:51.343192 2548 ts_tablet_manager.cc:1398] T b16391636d5f4ec48c9e4f18ed40badd P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:51.343894 30559 tablet_copy_service.cc:342] P 31811708c6774037b82df44024b229b6: Request end of tablet copy session aad81047009a4ffcbc450b60030583b5-b16391636d5f4ec48c9e4f18ed40badd received from {username='slave'} at 127.21.84.131:56895
I20240501 08:12:51.344060 30559 tablet_copy_service.cc:433] P 31811708c6774037b82df44024b229b6: ending tablet copy session aad81047009a4ffcbc450b60030583b5-b16391636d5f4ec48c9e4f18ed40badd on tablet b16391636d5f4ec48c9e4f18ed40badd with peer aad81047009a4ffcbc450b60030583b5
I20240501 08:12:51.347705 2364 ts_tablet_manager.cc:902] T 28a1f9ac749143de951c1b41beec9875 P 6331f77dae9d452184e8cd0cbf5f1b43: Initiating tablet copy from peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:51.348147 1772 raft_consensus.cc:1050] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5: attempting to promote NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 to VOTER
I20240501 08:12:51.348963 1772 consensus_queue.cc:237] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:12:51.351787 30536 raft_consensus.cc:1261] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:51.352023 30803 raft_consensus.cc:1261] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:51.352744 2231 consensus_queue.cc:1035] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:51.353271 2231 consensus_queue.cc:1035] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:51.353323 27759 raft_consensus.cc:1261] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:51.355316 2444 consensus_queue.cc:1035] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:51.361528 30537 consensus_queue.cc:237] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:51.374059 2364 tablet_copy_client.cc:308] T 28a1f9ac749143de951c1b41beec9875 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.134:39933
I20240501 08:12:51.377184 2216 raft_consensus.cc:1050] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5: attempting to promote NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 to VOTER
I20240501 08:12:51.378480 2216 consensus_queue.cc:237] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:12:51.386286 2231 raft_consensus.cc:2932] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:51.390494 30803 raft_consensus.cc:2932] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:51.395509 27759 raft_consensus.cc:2932] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:51.402719 27620 catalog_manager.cc:5484] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:51.406982 28149 raft_consensus.cc:1261] T c5f4c78e69fb4c67ad65a7b0c1466798 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:51.407575 28022 raft_consensus.cc:1261] T c5f4c78e69fb4c67ad65a7b0c1466798 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:51.409394 2216 consensus_queue.cc:1035] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:51.410526 2227 consensus_queue.cc:1035] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:51.415355 30536 raft_consensus.cc:1261] T c5f4c78e69fb4c67ad65a7b0c1466798 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:51.416409 27889 raft_consensus.cc:1203] T f607d2e086cc4352a05a5ef2b02788cc P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:51.421335 2229 consensus_queue.cc:1035] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:51.431980 28152 raft_consensus.cc:1203] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:51.434806 30693 tablet_copy_service.cc:140] P 9a022e1a501f4d44a0468054d6c3de47: Received BeginTabletCopySession request for tablet 28a1f9ac749143de951c1b41beec9875 from peer 6331f77dae9d452184e8cd0cbf5f1b43 ({username='slave'} at 127.21.84.129:59569)
I20240501 08:12:51.435043 30693 tablet_copy_service.cc:161] P 9a022e1a501f4d44a0468054d6c3de47: Beginning new tablet copy session on tablet 28a1f9ac749143de951c1b41beec9875 from peer 6331f77dae9d452184e8cd0cbf5f1b43 at {username='slave'} at 127.21.84.129:59569: session id = 6331f77dae9d452184e8cd0cbf5f1b43-28a1f9ac749143de951c1b41beec9875
I20240501 08:12:51.441037 30693 tablet_copy_source_session.cc:205] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:51.443472 2434 raft_consensus.cc:1050] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43: attempting to promote NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 to VOTER
I20240501 08:12:51.444341 2434 consensus_queue.cc:237] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:12:51.446768 2364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28a1f9ac749143de951c1b41beec9875. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:51.446534 2229 raft_consensus.cc:2932] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:51.453409 28020 raft_consensus.cc:2932] T c5f4c78e69fb4c67ad65a7b0c1466798 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:51.453408 28149 raft_consensus.cc:2932] T c5f4c78e69fb4c67ad65a7b0c1466798 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:51.461819 27620 catalog_manager.cc:5484] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:51.471005 30534 raft_consensus.cc:2932] T c5f4c78e69fb4c67ad65a7b0c1466798 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:51.500262 30670 raft_consensus.cc:1261] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:51.500803 30536 raft_consensus.cc:1261] T e94895331d6b474484d987d42f13fd88 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:51.501106 30800 raft_consensus.cc:1261] T e94895331d6b474484d987d42f13fd88 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:51.501631 1089 consensus_queue.cc:1035] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:51.508294 2364 tablet_copy_client.cc:783] T 28a1f9ac749143de951c1b41beec9875 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:51.509061 2364 tablet_copy_client.cc:647] T 28a1f9ac749143de951c1b41beec9875 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:51.510274 30533 raft_consensus.cc:2932] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:51.519079 2434 consensus_queue.cc:1035] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 08:12:51.519584 30473 consensus_peers.cc:485] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7affd631d1e444eaaf8a4c51267a204f. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:51.521777 30935 raft_consensus.cc:1261] T 7affd631d1e444eaaf8a4c51267a204f P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:51.522851 2580 consensus_queue.cc:1035] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:51.527559 2581 consensus_queue.cc:1035] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:51.528432 30670 raft_consensus.cc:1261] T 7affd631d1e444eaaf8a4c51267a204f P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:51.529134 2353 consensus_queue.cc:1035] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:51.540583 2364 tablet_copy_client.cc:523] T 28a1f9ac749143de951c1b41beec9875 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:51.545697 2364 tablet_bootstrap.cc:492] T 28a1f9ac749143de951c1b41beec9875 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:51.548622 1089 raft_consensus.cc:2932] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:51.548683 2353 raft_consensus.cc:2932] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:51.549249 30935 raft_consensus.cc:2932] T 7affd631d1e444eaaf8a4c51267a204f P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:51.562527 30672 raft_consensus.cc:2932] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:51.563788 27620 catalog_manager.cc:5484] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:51.578152 27620 catalog_manager.cc:5484] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 2 to 3, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:51.589520 30533 raft_consensus.cc:1261] T d3673fe03fb64c47b30dc2b79042f44a P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:12:51.590301 2123 consensus_queue.cc:1035] T d3673fe03fb64c47b30dc2b79042f44a P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:51.612870 30669 raft_consensus.cc:2932] T 7affd631d1e444eaaf8a4c51267a204f P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:51.621344 1740 raft_consensus.cc:1050] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6: attempting to promote NON_VOTER 64998323a1f4482d8704bf20594d13a5 to VOTER
I20240501 08:12:51.622181 1740 consensus_queue.cc:237] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:12:51.632208 2364 tablet_bootstrap.cc:492] T 28a1f9ac749143de951c1b41beec9875 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:51.632581 2364 tablet_bootstrap.cc:492] T 28a1f9ac749143de951c1b41beec9875 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap complete.
I20240501 08:12:51.632826 2364 ts_tablet_manager.cc:1367] T 28a1f9ac749143de951c1b41beec9875 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.087s user 0.007s sys 0.000s
I20240501 08:12:51.633493 2364 raft_consensus.cc:348] T 28a1f9ac749143de951c1b41beec9875 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:12:51.633718 2364 raft_consensus.cc:729] T 28a1f9ac749143de951c1b41beec9875 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: LEARNER
I20240501 08:12:51.633924 2364 consensus_queue.cc:260] T 28a1f9ac749143de951c1b41beec9875 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:12:51.639789 30800 raft_consensus.cc:1261] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:12:51.641819 2364 ts_tablet_manager.cc:1398] T 28a1f9ac749143de951c1b41beec9875 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240501 08:12:51.642532 2222 raft_consensus.cc:1050] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6: attempting to promote NON_VOTER a29cd590999b4f32849ff5689cd53c11 to VOTER
I20240501 08:12:51.643357 2222 consensus_queue.cc:237] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:12:51.644865 1378 consensus_queue.cc:1035] T d3673fe03fb64c47b30dc2b79042f44a P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:51.653189 30693 tablet_copy_service.cc:342] P 9a022e1a501f4d44a0468054d6c3de47: Request end of tablet copy session 6331f77dae9d452184e8cd0cbf5f1b43-28a1f9ac749143de951c1b41beec9875 received from {username='slave'} at 127.21.84.129:59569
I20240501 08:12:51.653383 30693 tablet_copy_service.cc:433] P 9a022e1a501f4d44a0468054d6c3de47: ending tablet copy session 6331f77dae9d452184e8cd0cbf5f1b43-28a1f9ac749143de951c1b41beec9875 on tablet 28a1f9ac749143de951c1b41beec9875 with peer 6331f77dae9d452184e8cd0cbf5f1b43
W20240501 08:12:51.670025 2584 log.cc:924] Time spent T e94895331d6b474484d987d42f13fd88 P 64998323a1f4482d8704bf20594d13a5: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20240501 08:12:51.670107 27889 consensus_queue.cc:237] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:12:51.676780 30803 raft_consensus.cc:2932] T e94895331d6b474484d987d42f13fd88 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:51.678822 30534 raft_consensus.cc:2932] T e94895331d6b474484d987d42f13fd88 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:51.727403 27760 consensus_queue.cc:237] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:12:51.731329 30670 raft_consensus.cc:1261] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:51.731913 2592 consensus_queue.cc:1035] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:51.734968 2580 raft_consensus.cc:2932] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:51.738909 30670 raft_consensus.cc:2932] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:51.746904 30801 raft_consensus.cc:1261] T e94895331d6b474484d987d42f13fd88 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:51.747064 27612 catalog_manager.cc:5484] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:51.749908 1976 raft_consensus.cc:1050] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47: attempting to promote NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 to VOTER
I20240501 08:12:51.751873 1976 consensus_queue.cc:237] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:12:51.753283 30514 tablet_service.cc:1514] Processing DeleteTablet for tablet e94895331d6b474484d987d42f13fd88 with delete_type TABLET_DATA_TOMBSTONED (TS 31811708c6774037b82df44024b229b6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45078
I20240501 08:12:51.764864 2580 consensus_queue.cc:1035] T e94895331d6b474484d987d42f13fd88 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:51.766721 30803 raft_consensus.cc:2932] T e94895331d6b474484d987d42f13fd88 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:51.777683 28022 raft_consensus.cc:1203] T b16391636d5f4ec48c9e4f18ed40badd P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:51.812230 30803 raft_consensus.cc:1261] T a0f992f08fb841a5b18b14f5ad0b9c58 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:51.812773 2607 consensus_queue.cc:1035] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 08:12:51.814249 27827 consensus_peers.cc:485] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af -> Peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): Couldn't send request to peer a29cd590999b4f32849ff5689cd53c11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 367847f920184dd6abcdcd7564f76376. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:51.815346 28022 raft_consensus.cc:1261] T 367847f920184dd6abcdcd7564f76376 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:51.815935 2531 consensus_queue.cc:1035] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:51.816457 2610 tablet_replica.cc:330] T e94895331d6b474484d987d42f13fd88 P 31811708c6774037b82df44024b229b6: stopping tablet replica
I20240501 08:12:51.816730 2610 raft_consensus.cc:2229] T e94895331d6b474484d987d42f13fd88 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:51.817004 2610 raft_consensus.cc:2258] T e94895331d6b474484d987d42f13fd88 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:51.818218 2610 ts_tablet_manager.cc:1875] T e94895331d6b474484d987d42f13fd88 P 31811708c6774037b82df44024b229b6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:51.818480 28152 raft_consensus.cc:1261] T 367847f920184dd6abcdcd7564f76376 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:51.818953 2531 consensus_queue.cc:1035] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:51.829257 2216 raft_consensus.cc:1050] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5: attempting to promote NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 to VOTER
W20240501 08:12:51.829286 28088 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.130:36503, user_credentials={real_user=slave}} blocked reactor thread for 123023us
I20240501 08:12:51.830276 2216 consensus_queue.cc:237] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:12:51.830651 27892 raft_consensus.cc:1261] T a0f992f08fb841a5b18b14f5ad0b9c58 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:51.831825 2610 ts_tablet_manager.cc:1888] T e94895331d6b474484d987d42f13fd88 P 31811708c6774037b82df44024b229b6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:51.831956 2610 log.cc:1196] T e94895331d6b474484d987d42f13fd88 P 31811708c6774037b82df44024b229b6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/wals/e94895331d6b474484d987d42f13fd88
I20240501 08:12:51.833288 27607 catalog_manager.cc:4830] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): tablet e94895331d6b474484d987d42f13fd88 (table rebalancer_extra_table_8 [id=f51464bcf46d406a807b68e0429a75af]) successfully deleted
I20240501 08:12:51.834239 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e94895331d6b474484d987d42f13fd88 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:51.836092 28149 raft_consensus.cc:1261] T d3673fe03fb64c47b30dc2b79042f44a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:12:51.841534 27760 raft_consensus.cc:1261] T 00a706ff4ea7456aacd2586791704e35 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:51.841789 28022 raft_consensus.cc:1261] T 00a706ff4ea7456aacd2586791704e35 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:51.843519 2436 consensus_queue.cc:1035] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:51.843680 30534 raft_consensus.cc:1261] T 00a706ff4ea7456aacd2586791704e35 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:51.844105 2436 consensus_queue.cc:1035] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:51.844797 2436 consensus_queue.cc:1035] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:51.847373 2531 raft_consensus.cc:2932] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:51.853405 28151 raft_consensus.cc:1261] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:51.855316 2226 consensus_queue.cc:1035] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:51.856705 28020 raft_consensus.cc:1261] T 9df818d6cb4946989f8a4d2e2ce9b8bb P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:51.857633 2226 consensus_queue.cc:1035] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:51.857774 30533 raft_consensus.cc:1261] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:51.858817 2216 consensus_queue.cc:1035] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:51.864424 28020 raft_consensus.cc:2932] T 367847f920184dd6abcdcd7564f76376 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:51.874060 2602 consensus_queue.cc:1035] T d3673fe03fb64c47b30dc2b79042f44a P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:51.876060 28149 raft_consensus.cc:2932] T 367847f920184dd6abcdcd7564f76376 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:51.879451 2226 raft_consensus.cc:2932] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:51.880534 27615 catalog_manager.cc:5484] T 367847f920184dd6abcdcd7564f76376 P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index -1 to 4, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:51.882869 30533 raft_consensus.cc:2932] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:51.887929 28151 raft_consensus.cc:2932] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:51.897019 27612 catalog_manager.cc:5484] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:51.905845 30935 raft_consensus.cc:1261] T a05223cb2f20478b8d29ffa065b4c67b P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:51.906066 27758 raft_consensus.cc:1261] T a05223cb2f20478b8d29ffa065b4c67b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:51.907001 28016 raft_consensus.cc:1261] T a05223cb2f20478b8d29ffa065b4c67b P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:51.907466 30803 consensus_queue.cc:237] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:12:51.908043 28016 raft_consensus.cc:1261] T a0f992f08fb841a5b18b14f5ad0b9c58 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:51.909682 28019 raft_consensus.cc:1261] T 9df818d6cb4946989f8a4d2e2ce9b8bb P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:51.911762 2216 consensus_queue.cc:1035] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:51.912930 2609 consensus_queue.cc:1035] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:51.914023 28016 raft_consensus.cc:2932] T 9df818d6cb4946989f8a4d2e2ce9b8bb P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:51.924657 28152 raft_consensus.cc:1261] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:51.925611 2216 consensus_queue.cc:1035] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:12:51.927721 28088 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.130:36503, user_credentials={real_user=slave}} blocked reactor thread for 54128.7us
I20240501 08:12:51.928419 2216 raft_consensus.cc:2932] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:51.931313 2227 raft_consensus.cc:1050] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5: attempting to promote NON_VOTER f1525cbc82a24d13b65f136695c903af to VOTER
I20240501 08:12:51.932252 27605 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9df818d6cb4946989f8a4d2e2ce9b8bb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:51.932161 2227 consensus_queue.cc:237] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:51.933111 28152 raft_consensus.cc:2932] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:51.938623 28020 raft_consensus.cc:2932] T 9df818d6cb4946989f8a4d2e2ce9b8bb P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:51.942097 27620 catalog_manager.cc:5484] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:51.946368 30514 tablet_service.cc:1514] Processing DeleteTablet for tablet 9df818d6cb4946989f8a4d2e2ce9b8bb with delete_type TABLET_DATA_TOMBSTONED (TS 31811708c6774037b82df44024b229b6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45078
I20240501 08:12:51.946787 2610 tablet_replica.cc:330] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 31811708c6774037b82df44024b229b6: stopping tablet replica
I20240501 08:12:51.947016 2610 raft_consensus.cc:2229] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:51.947314 2610 raft_consensus.cc:2258] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:51.948310 2610 ts_tablet_manager.cc:1875] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 31811708c6774037b82df44024b229b6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:51.955348 2610 ts_tablet_manager.cc:1888] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 31811708c6774037b82df44024b229b6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:51.955583 2610 log.cc:1196] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 31811708c6774037b82df44024b229b6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/wals/9df818d6cb4946989f8a4d2e2ce9b8bb
I20240501 08:12:51.955983 30669 raft_consensus.cc:1261] T f607d2e086cc4352a05a5ef2b02788cc P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:51.956657 2216 consensus_queue.cc:1035] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:51.956915 27892 raft_consensus.cc:1261] T f607d2e086cc4352a05a5ef2b02788cc P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:51.959086 27758 raft_consensus.cc:1203] T 28a1f9ac749143de951c1b41beec9875 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:51.959681 2216 consensus_queue.cc:1035] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:51.956545 27607 catalog_manager.cc:4830] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): tablet 9df818d6cb4946989f8a4d2e2ce9b8bb (table rebalancer_extra_table_4 [id=0d602f5b725a462185dafa660a8e4c1a]) successfully deleted
W20240501 08:12:51.989238 30870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.132:41323, user_credentials={real_user=slave}} blocked reactor thread for 115270us
I20240501 08:12:51.989545 1679 raft_consensus.cc:2932] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:51.994783 30935 raft_consensus.cc:1261] T f607d2e086cc4352a05a5ef2b02788cc P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:51.996395 2227 consensus_queue.cc:1035] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:52.001238 27615 catalog_manager.cc:5484] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:52.007262 27758 raft_consensus.cc:2932] T 00a706ff4ea7456aacd2586791704e35 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:52.014307 28022 raft_consensus.cc:2932] T 00a706ff4ea7456aacd2586791704e35 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:52.033592 2609 consensus_queue.cc:1035] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:52.037067 2463 raft_consensus.cc:2932] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:52.047003 27891 raft_consensus.cc:2932] T a0f992f08fb841a5b18b14f5ad0b9c58 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:52.047665 30800 raft_consensus.cc:2932] T a0f992f08fb841a5b18b14f5ad0b9c58 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:52.053954 2634 ts_tablet_manager.cc:902] T 7affd631d1e444eaaf8a4c51267a204f P f1525cbc82a24d13b65f136695c903af: Initiating tablet copy from peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:52.082366 2227 raft_consensus.cc:2932] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:52.086238 2609 consensus_queue.cc:1035] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:52.088874 30935 raft_consensus.cc:2932] T f607d2e086cc4352a05a5ef2b02788cc P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:52.089735 30672 raft_consensus.cc:2932] T f607d2e086cc4352a05a5ef2b02788cc P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:52.095016 2606 consensus_queue.cc:1035] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:52.097851 27612 catalog_manager.cc:5484] T f607d2e086cc4352a05a5ef2b02788cc P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:52.099354 2606 consensus_queue.cc:1035] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:52.113343 27615 catalog_manager.cc:5484] T a0f992f08fb841a5b18b14f5ad0b9c58 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:52.112736 27892 raft_consensus.cc:2932] T f607d2e086cc4352a05a5ef2b02788cc P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:52.126863 30800 consensus_queue.cc:237] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:52.130735 30934 raft_consensus.cc:1261] T f607d2e086cc4352a05a5ef2b02788cc P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:52.131383 2227 consensus_queue.cc:1035] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:52.134593 30934 raft_consensus.cc:2932] T f607d2e086cc4352a05a5ef2b02788cc P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:52.133872 2227 raft_consensus.cc:2932] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:52.146090 27605 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f607d2e086cc4352a05a5ef2b02788cc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:52.154474 30672 consensus_queue.cc:237] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:12:52.154708 27620 catalog_manager.cc:5484] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:52.156505 30536 raft_consensus.cc:1261] T 00a706ff4ea7456aacd2586791704e35 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:52.157016 27758 raft_consensus.cc:1261] T 00a706ff4ea7456aacd2586791704e35 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:52.158102 27889 raft_consensus.cc:1261] T f607d2e086cc4352a05a5ef2b02788cc P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:52.158342 2436 consensus_queue.cc:1035] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:52.158787 2436 consensus_queue.cc:1035] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:52.161859 2216 consensus_queue.cc:1035] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:52.161813 30536 raft_consensus.cc:2932] T 00a706ff4ea7456aacd2586791704e35 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:52.166764 1679 raft_consensus.cc:2932] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:52.166672 27892 raft_consensus.cc:2932] T f607d2e086cc4352a05a5ef2b02788cc P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:52.171164 27759 raft_consensus.cc:2932] T 00a706ff4ea7456aacd2586791704e35 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:52.179113 27604 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 00a706ff4ea7456aacd2586791704e35 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:52.179041 27611 catalog_manager.cc:5484] T 00a706ff4ea7456aacd2586791704e35 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index 3 to 4, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New cstate: current_term: 1 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:52.185357 28000 tablet_service.cc:1514] Processing DeleteTablet for tablet 00a706ff4ea7456aacd2586791704e35 with delete_type TABLET_DATA_TOMBSTONED (TS aad81047009a4ffcbc450b60030583b5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:52.194820 30536 raft_consensus.cc:2932] T 00a706ff4ea7456aacd2586791704e35 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:52.196689 30650 tablet_service.cc:1514] Processing DeleteTablet for tablet f607d2e086cc4352a05a5ef2b02788cc with delete_type TABLET_DATA_TOMBSTONED (TS 9a022e1a501f4d44a0468054d6c3de47 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57156
I20240501 08:12:52.204519 2647 tablet_replica.cc:330] T f607d2e086cc4352a05a5ef2b02788cc P 9a022e1a501f4d44a0468054d6c3de47: stopping tablet replica
I20240501 08:12:52.204845 2647 raft_consensus.cc:2229] T f607d2e086cc4352a05a5ef2b02788cc P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:52.205122 2647 raft_consensus.cc:2258] T f607d2e086cc4352a05a5ef2b02788cc P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:52.206202 2647 ts_tablet_manager.cc:1875] T f607d2e086cc4352a05a5ef2b02788cc P 9a022e1a501f4d44a0468054d6c3de47: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:52.225246 2647 ts_tablet_manager.cc:1888] T f607d2e086cc4352a05a5ef2b02788cc P 9a022e1a501f4d44a0468054d6c3de47: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:52.225589 2647 log.cc:1196] T f607d2e086cc4352a05a5ef2b02788cc P 9a022e1a501f4d44a0468054d6c3de47: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/wals/f607d2e086cc4352a05a5ef2b02788cc
I20240501 08:12:52.229002 27604 catalog_manager.cc:4830] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): tablet f607d2e086cc4352a05a5ef2b02788cc (table rebalancer_extra_table_22 [id=dbd065db73d1428c8fe77bd9af3302f6]) successfully deleted
I20240501 08:12:52.238607 2648 ts_tablet_manager.cc:902] T 367847f920184dd6abcdcd7564f76376 P a29cd590999b4f32849ff5689cd53c11: Initiating tablet copy from peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:12:52.244570 30934 consensus_queue.cc:237] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:12:52.245517 2646 tablet_replica.cc:330] T 00a706ff4ea7456aacd2586791704e35 P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:52.245815 2646 raft_consensus.cc:2229] T 00a706ff4ea7456aacd2586791704e35 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:52.246160 2646 raft_consensus.cc:2258] T 00a706ff4ea7456aacd2586791704e35 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:52.248417 2634 tablet_copy_client.cc:308] T 7affd631d1e444eaaf8a4c51267a204f P f1525cbc82a24d13b65f136695c903af: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.133:38293
I20240501 08:12:52.248893 2646 ts_tablet_manager.cc:1875] T 00a706ff4ea7456aacd2586791704e35 P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:52.250917 30559 tablet_copy_service.cc:140] P 31811708c6774037b82df44024b229b6: Received BeginTabletCopySession request for tablet 7affd631d1e444eaaf8a4c51267a204f from peer f1525cbc82a24d13b65f136695c903af ({username='slave'} at 127.21.84.130:58953)
I20240501 08:12:52.251122 30559 tablet_copy_service.cc:161] P 31811708c6774037b82df44024b229b6: Beginning new tablet copy session on tablet 7affd631d1e444eaaf8a4c51267a204f from peer f1525cbc82a24d13b65f136695c903af at {username='slave'} at 127.21.84.130:58953: session id = f1525cbc82a24d13b65f136695c903af-7affd631d1e444eaaf8a4c51267a204f
I20240501 08:12:52.254069 30559 tablet_copy_source_session.cc:205] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:52.254828 2646 ts_tablet_manager.cc:1888] T 00a706ff4ea7456aacd2586791704e35 P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:52.254997 2646 log.cc:1196] T 00a706ff4ea7456aacd2586791704e35 P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/00a706ff4ea7456aacd2586791704e35
I20240501 08:12:52.255901 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet 00a706ff4ea7456aacd2586791704e35 (table rebalancer_extra_table_4 [id=0d602f5b725a462185dafa660a8e4c1a]) successfully deleted
I20240501 08:12:52.258196 2634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7affd631d1e444eaaf8a4c51267a204f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:52.263062 2634 tablet_copy_client.cc:783] T 7affd631d1e444eaaf8a4c51267a204f P f1525cbc82a24d13b65f136695c903af: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:52.263335 2634 tablet_copy_client.cc:647] T 7affd631d1e444eaaf8a4c51267a204f P f1525cbc82a24d13b65f136695c903af: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:52.266176 2634 tablet_copy_client.cc:523] T 7affd631d1e444eaaf8a4c51267a204f P f1525cbc82a24d13b65f136695c903af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:52.269578 2634 tablet_bootstrap.cc:492] T 7affd631d1e444eaaf8a4c51267a204f P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:52.275269 2634 tablet_bootstrap.cc:492] T 7affd631d1e444eaaf8a4c51267a204f P f1525cbc82a24d13b65f136695c903af: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:52.275693 2634 tablet_bootstrap.cc:492] T 7affd631d1e444eaaf8a4c51267a204f P f1525cbc82a24d13b65f136695c903af: Bootstrap complete.
I20240501 08:12:52.275904 2634 ts_tablet_manager.cc:1367] T 7affd631d1e444eaaf8a4c51267a204f P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20240501 08:12:52.276516 2634 raft_consensus.cc:348] T 7affd631d1e444eaaf8a4c51267a204f P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:52.276685 2634 raft_consensus.cc:729] T 7affd631d1e444eaaf8a4c51267a204f P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: LEARNER
I20240501 08:12:52.276855 2634 consensus_queue.cc:260] T 7affd631d1e444eaaf8a4c51267a204f P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:52.277464 2634 ts_tablet_manager.cc:1398] T 7affd631d1e444eaaf8a4c51267a204f P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 08:12:52.278793 30559 tablet_copy_service.cc:342] P 31811708c6774037b82df44024b229b6: Request end of tablet copy session f1525cbc82a24d13b65f136695c903af-7affd631d1e444eaaf8a4c51267a204f received from {username='slave'} at 127.21.84.130:58953
I20240501 08:12:52.278888 30559 tablet_copy_service.cc:433] P 31811708c6774037b82df44024b229b6: ending tablet copy session f1525cbc82a24d13b65f136695c903af-7affd631d1e444eaaf8a4c51267a204f on tablet 7affd631d1e444eaaf8a4c51267a204f with peer f1525cbc82a24d13b65f136695c903af
W20240501 08:12:52.280606 30872 consensus_peers.cc:485] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f63215605ade41a5bc8d97a35ce33a31. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:52.282475 2222 raft_consensus.cc:2932] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:52.283389 28020 raft_consensus.cc:2932] T a05223cb2f20478b8d29ffa065b4c67b P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:52.284435 30534 raft_consensus.cc:1261] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:52.285041 2589 consensus_queue.cc:1035] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:52.288012 30935 raft_consensus.cc:2932] T a05223cb2f20478b8d29ffa065b4c67b P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:52.292312 27611 catalog_manager.cc:5484] T a05223cb2f20478b8d29ffa065b4c67b P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:52.303802 28020 raft_consensus.cc:1261] T f63215605ade41a5bc8d97a35ce33a31 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:52.303612 28022 raft_consensus.cc:2932] T a0f992f08fb841a5b18b14f5ad0b9c58 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:52.305840 2589 consensus_queue.cc:1035] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:52.306468 27759 raft_consensus.cc:2932] T a05223cb2f20478b8d29ffa065b4c67b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:52.311417 2594 raft_consensus.cc:1050] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6: attempting to promote NON_VOTER aad81047009a4ffcbc450b60030583b5 to VOTER
I20240501 08:12:52.312413 2594 consensus_queue.cc:237] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } }
I20240501 08:12:52.319963 2602 raft_consensus.cc:2932] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:52.322993 30533 raft_consensus.cc:2932] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:52.332798 27620 catalog_manager.cc:5484] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:52.340543 30800 raft_consensus.cc:1261] T b16391636d5f4ec48c9e4f18ed40badd P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:52.343899 28018 raft_consensus.cc:1261] T b16391636d5f4ec48c9e4f18ed40badd P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:52.344496 2422 consensus_queue.cc:1035] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:52.344626 27759 raft_consensus.cc:1261] T b16391636d5f4ec48c9e4f18ed40badd P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:52.345118 2422 consensus_queue.cc:1035] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:52.348357 2422 consensus_queue.cc:1035] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:52.351260 28016 raft_consensus.cc:2932] T f63215605ade41a5bc8d97a35ce33a31 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:52.369437 2648 tablet_copy_client.cc:308] T 367847f920184dd6abcdcd7564f76376 P a29cd590999b4f32849ff5689cd53c11: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.130:36503
I20240501 08:12:52.370345 27912 tablet_copy_service.cc:140] P f1525cbc82a24d13b65f136695c903af: Received BeginTabletCopySession request for tablet 367847f920184dd6abcdcd7564f76376 from peer a29cd590999b4f32849ff5689cd53c11 ({username='slave'} at 127.21.84.136:54747)
I20240501 08:12:52.370517 27912 tablet_copy_service.cc:161] P f1525cbc82a24d13b65f136695c903af: Beginning new tablet copy session on tablet 367847f920184dd6abcdcd7564f76376 from peer a29cd590999b4f32849ff5689cd53c11 at {username='slave'} at 127.21.84.136:54747: session id = a29cd590999b4f32849ff5689cd53c11-367847f920184dd6abcdcd7564f76376
W20240501 08:12:52.370784 2650 log.cc:924] Time spent T a0f992f08fb841a5b18b14f5ad0b9c58 P aad81047009a4ffcbc450b60030583b5: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20240501 08:12:52.374953 27912 tablet_copy_source_session.cc:205] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:52.380964 2648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 367847f920184dd6abcdcd7564f76376. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:52.392045 2422 raft_consensus.cc:2932] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:52.396387 30803 raft_consensus.cc:2932] T b16391636d5f4ec48c9e4f18ed40badd P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:52.405187 27760 raft_consensus.cc:2932] T b16391636d5f4ec48c9e4f18ed40badd P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:52.406585 27612 catalog_manager.cc:5484] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:52.397156 28018 raft_consensus.cc:2932] T b16391636d5f4ec48c9e4f18ed40badd P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:52.417058 30534 consensus_queue.cc:237] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } }
I20240501 08:12:52.422286 28018 raft_consensus.cc:1261] T b16391636d5f4ec48c9e4f18ed40badd P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:52.424109 2594 consensus_queue.cc:1035] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:52.432371 27759 raft_consensus.cc:1261] T b16391636d5f4ec48c9e4f18ed40badd P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:52.434864 30936 consensus_queue.cc:237] T d3673fe03fb64c47b30dc2b79042f44a P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:12:52.439587 28152 raft_consensus.cc:1261] T d3673fe03fb64c47b30dc2b79042f44a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:12:52.441054 30534 raft_consensus.cc:1261] T d3673fe03fb64c47b30dc2b79042f44a P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:12:52.442437 2602 consensus_queue.cc:1035] T d3673fe03fb64c47b30dc2b79042f44a P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:52.444528 1378 consensus_queue.cc:1035] T d3673fe03fb64c47b30dc2b79042f44a P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:52.458324 1378 raft_consensus.cc:2932] T d3673fe03fb64c47b30dc2b79042f44a P a29cd590999b4f32849ff5689cd53c11 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:52.463004 28152 raft_consensus.cc:2932] T d3673fe03fb64c47b30dc2b79042f44a P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:52.463800 30533 raft_consensus.cc:2932] T d3673fe03fb64c47b30dc2b79042f44a P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
W20240501 08:12:52.471604 27697 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.133:38293, user_credentials={real_user=slave}} blocked reactor thread for 46951.1us
I20240501 08:12:52.475241 2422 consensus_queue.cc:1035] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 08:12:52.475814 2422 raft_consensus.cc:2932] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:52.477978 28022 raft_consensus.cc:2932] T b16391636d5f4ec48c9e4f18ed40badd P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:52.479806 27607 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b16391636d5f4ec48c9e4f18ed40badd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:52.481825 27620 catalog_manager.cc:5484] T d3673fe03fb64c47b30dc2b79042f44a P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index 3 to 5, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New cstate: current_term: 2 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:52.486078 30784 tablet_service.cc:1514] Processing DeleteTablet for tablet d3673fe03fb64c47b30dc2b79042f44a with delete_type TABLET_DATA_TOMBSTONED (TS 64998323a1f4482d8704bf20594d13a5 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:38952
I20240501 08:12:52.488373 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d3673fe03fb64c47b30dc2b79042f44a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:52.490401 1679 raft_consensus.cc:1050] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47: attempting to promote NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 to VOTER
I20240501 08:12:52.491466 1679 consensus_queue.cc:237] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:12:52.493474 27760 raft_consensus.cc:2932] T b16391636d5f4ec48c9e4f18ed40badd P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:52.496657 27612 catalog_manager.cc:5484] T b16391636d5f4ec48c9e4f18ed40badd P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 3 to 4, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:52.502094 30784 tablet_service.cc:1514] Processing DeleteTablet for tablet b16391636d5f4ec48c9e4f18ed40badd with delete_type TABLET_DATA_TOMBSTONED (TS 64998323a1f4482d8704bf20594d13a5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38952
I20240501 08:12:52.511044 27760 raft_consensus.cc:1261] T 28a1f9ac749143de951c1b41beec9875 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:52.511595 1679 consensus_queue.cc:1035] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:52.512709 30800 raft_consensus.cc:1261] T 28a1f9ac749143de951c1b41beec9875 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:52.513216 30934 raft_consensus.cc:1261] T 28a1f9ac749143de951c1b41beec9875 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:52.514205 1976 consensus_queue.cc:1035] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:52.514741 1976 consensus_queue.cc:1035] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:52.519171 2679 tablet_replica.cc:330] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5: stopping tablet replica
I20240501 08:12:52.519430 2679 raft_consensus.cc:2229] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:52.520714 2679 raft_consensus.cc:2258] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:52.521746 2679 ts_tablet_manager.cc:1875] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:52.524300 2648 tablet_copy_client.cc:783] T 367847f920184dd6abcdcd7564f76376 P a29cd590999b4f32849ff5689cd53c11: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:52.524832 2648 tablet_copy_client.cc:647] T 367847f920184dd6abcdcd7564f76376 P a29cd590999b4f32849ff5689cd53c11: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:52.530323 28018 consensus_queue.cc:237] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:52.534386 27892 raft_consensus.cc:1203] T 7affd631d1e444eaaf8a4c51267a204f P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:52.537380 2648 tablet_copy_client.cc:523] T 367847f920184dd6abcdcd7564f76376 P a29cd590999b4f32849ff5689cd53c11: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:52.543459 1976 raft_consensus.cc:2932] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:52.551541 30803 raft_consensus.cc:2932] T 28a1f9ac749143de951c1b41beec9875 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:52.553213 30670 raft_consensus.cc:1261] T 87bd346903dc4c2c85ecaf362fe57799 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:52.553745 1772 consensus_queue.cc:1035] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:52.555369 30934 raft_consensus.cc:2932] T 28a1f9ac749143de951c1b41beec9875 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
W20240501 08:12:52.558763 27958 consensus_peers.cc:485] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87bd346903dc4c2c85ecaf362fe57799. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:52.559965 30533 raft_consensus.cc:1261] T 87bd346903dc4c2c85ecaf362fe57799 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:52.560518 1772 consensus_queue.cc:1035] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:52.564667 1772 raft_consensus.cc:2932] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:52.569548 30672 raft_consensus.cc:2932] T 87bd346903dc4c2c85ecaf362fe57799 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:52.576436 30533 raft_consensus.cc:2932] T 87bd346903dc4c2c85ecaf362fe57799 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:52.576984 27612 catalog_manager.cc:5484] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:52.578428 27760 raft_consensus.cc:2932] T 28a1f9ac749143de951c1b41beec9875 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:52.581629 30672 consensus_queue.cc:237] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:12:52.588773 27612 catalog_manager.cc:5484] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:52.590723 27759 raft_consensus.cc:1261] T 28a1f9ac749143de951c1b41beec9875 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:52.591493 2436 consensus_queue.cc:1035] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:52.598359 30800 raft_consensus.cc:1261] T 28a1f9ac749143de951c1b41beec9875 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:52.599700 1679 consensus_queue.cc:1035] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:52.600050 2648 tablet_bootstrap.cc:492] T 367847f920184dd6abcdcd7564f76376 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:52.602202 2436 raft_consensus.cc:2932] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:52.606231 27604 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 28a1f9ac749143de951c1b41beec9875 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:52.610620 27620 catalog_manager.cc:5484] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New cstate: current_term: 1 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:52.615156 30916 tablet_service.cc:1514] Processing DeleteTablet for tablet 28a1f9ac749143de951c1b41beec9875 with delete_type TABLET_DATA_TOMBSTONED (TS a29cd590999b4f32849ff5689cd53c11 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54036
I20240501 08:12:52.616667 27759 raft_consensus.cc:2932] T 28a1f9ac749143de951c1b41beec9875 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:52.620669 2690 tablet_replica.cc:330] T 28a1f9ac749143de951c1b41beec9875 P a29cd590999b4f32849ff5689cd53c11: stopping tablet replica
I20240501 08:12:52.621418 2690 raft_consensus.cc:2229] T 28a1f9ac749143de951c1b41beec9875 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:52.624927 2690 raft_consensus.cc:2258] T 28a1f9ac749143de951c1b41beec9875 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:52.626317 2679 ts_tablet_manager.cc:1888] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240501 08:12:52.627596 2690 ts_tablet_manager.cc:1875] T 28a1f9ac749143de951c1b41beec9875 P a29cd590999b4f32849ff5689cd53c11: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:52.631762 2679 log.cc:1196] T d3673fe03fb64c47b30dc2b79042f44a P 64998323a1f4482d8704bf20594d13a5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal/wals/d3673fe03fb64c47b30dc2b79042f44a
I20240501 08:12:52.632854 2679 tablet_replica.cc:330] T b16391636d5f4ec48c9e4f18ed40badd P 64998323a1f4482d8704bf20594d13a5: stopping tablet replica
I20240501 08:12:52.632844 27605 catalog_manager.cc:4830] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): tablet d3673fe03fb64c47b30dc2b79042f44a (table rebalancer_extra_table_42 [id=6aa96b40a8e74b81bfc5ccab950a210d]) successfully deleted
I20240501 08:12:52.630668 30800 raft_consensus.cc:2932] T 28a1f9ac749143de951c1b41beec9875 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:52.645530 2679 raft_consensus.cc:2229] T b16391636d5f4ec48c9e4f18ed40badd P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:52.646174 2679 raft_consensus.cc:2258] T b16391636d5f4ec48c9e4f18ed40badd P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:52.652446 2679 ts_tablet_manager.cc:1875] T b16391636d5f4ec48c9e4f18ed40badd P 64998323a1f4482d8704bf20594d13a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:52.663208 2690 ts_tablet_manager.cc:1888] T 28a1f9ac749143de951c1b41beec9875 P a29cd590999b4f32849ff5689cd53c11: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:52.670871 2690 log.cc:1196] T 28a1f9ac749143de951c1b41beec9875 P a29cd590999b4f32849ff5689cd53c11: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/wals/28a1f9ac749143de951c1b41beec9875
I20240501 08:12:52.670393 2648 tablet_bootstrap.cc:492] T 367847f920184dd6abcdcd7564f76376 P a29cd590999b4f32849ff5689cd53c11: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:52.671414 2648 tablet_bootstrap.cc:492] T 367847f920184dd6abcdcd7564f76376 P a29cd590999b4f32849ff5689cd53c11: Bootstrap complete.
I20240501 08:12:52.671730 27606 catalog_manager.cc:4830] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): tablet 28a1f9ac749143de951c1b41beec9875 (table rebalancer_extra_table_54 [id=20ee5c0014e146438233fd009facbf2c]) successfully deleted
I20240501 08:12:52.671994 2648 ts_tablet_manager.cc:1367] T 367847f920184dd6abcdcd7564f76376 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.072s user 0.051s sys 0.012s
I20240501 08:12:52.673120 2679 ts_tablet_manager.cc:1888] T b16391636d5f4ec48c9e4f18ed40badd P 64998323a1f4482d8704bf20594d13a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:52.673250 2679 log.cc:1196] T b16391636d5f4ec48c9e4f18ed40badd P 64998323a1f4482d8704bf20594d13a5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal/wals/b16391636d5f4ec48c9e4f18ed40badd
I20240501 08:12:52.673403 2648 raft_consensus.cc:348] T 367847f920184dd6abcdcd7564f76376 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:12:52.673743 2648 raft_consensus.cc:729] T 367847f920184dd6abcdcd7564f76376 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: LEARNER
I20240501 08:12:52.674122 2648 consensus_queue.cc:260] T 367847f920184dd6abcdcd7564f76376 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:12:52.679495 27605 catalog_manager.cc:4830] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): tablet b16391636d5f4ec48c9e4f18ed40badd (table rebalancer_extra_table_46 [id=ec67e56c6aaa43f3b2e573534276d8bc]) successfully deleted
I20240501 08:12:52.684404 2648 ts_tablet_manager.cc:1398] T 367847f920184dd6abcdcd7564f76376 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240501 08:12:52.690586 27912 tablet_copy_service.cc:342] P f1525cbc82a24d13b65f136695c903af: Request end of tablet copy session a29cd590999b4f32849ff5689cd53c11-367847f920184dd6abcdcd7564f76376 received from {username='slave'} at 127.21.84.136:54747
I20240501 08:12:52.690729 27912 tablet_copy_service.cc:433] P f1525cbc82a24d13b65f136695c903af: ending tablet copy session a29cd590999b4f32849ff5689cd53c11-367847f920184dd6abcdcd7564f76376 on tablet 367847f920184dd6abcdcd7564f76376 with peer a29cd590999b4f32849ff5689cd53c11
I20240501 08:12:52.692739 28152 consensus_queue.cc:237] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:12:52.694965 30800 raft_consensus.cc:1261] T a0f992f08fb841a5b18b14f5ad0b9c58 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:52.695811 2222 consensus_queue.cc:1035] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:52.698904 28022 raft_consensus.cc:1261] T a0f992f08fb841a5b18b14f5ad0b9c58 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:52.699705 2463 consensus_queue.cc:1035] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:52.700578 2607 raft_consensus.cc:2932] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:52.708252 30800 raft_consensus.cc:2932] T a0f992f08fb841a5b18b14f5ad0b9c58 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:52.714493 28022 raft_consensus.cc:2932] T a0f992f08fb841a5b18b14f5ad0b9c58 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:52.725311 27759 consensus_queue.cc:237] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:12:52.732239 27604 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a0f992f08fb841a5b18b14f5ad0b9c58 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:52.735092 27611 catalog_manager.cc:5484] T a0f992f08fb841a5b18b14f5ad0b9c58 P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 5 to 6, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:52.738852 30803 raft_consensus.cc:1261] T 6a5f99276dad456ea80e3c9f9618059a P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:52.740576 27892 raft_consensus.cc:1261] T 6a5f99276dad456ea80e3c9f9618059a P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:52.741169 2674 consensus_queue.cc:1035] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:12:52.742015 27696 consensus_peers.cc:485] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 -> Peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Couldn't send request to peer aad81047009a4ffcbc450b60030583b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a5f99276dad456ea80e3c9f9618059a. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:52.742586 2674 consensus_queue.cc:1035] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:52.758595 2673 raft_consensus.cc:2932] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:52.759963 27871 tablet_service.cc:1514] Processing DeleteTablet for tablet a0f992f08fb841a5b18b14f5ad0b9c58 with delete_type TABLET_DATA_TOMBSTONED (TS f1525cbc82a24d13b65f136695c903af not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:52.760548 30934 raft_consensus.cc:1203] T 367847f920184dd6abcdcd7564f76376 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:52.764377 30800 raft_consensus.cc:2932] T 6a5f99276dad456ea80e3c9f9618059a P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:52.771978 27892 raft_consensus.cc:2932] T 6a5f99276dad456ea80e3c9f9618059a P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:52.776849 27612 catalog_manager.cc:5484] T 6a5f99276dad456ea80e3c9f9618059a P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:52.778517 2698 tablet_replica.cc:330] T a0f992f08fb841a5b18b14f5ad0b9c58 P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:52.778782 2698 raft_consensus.cc:2229] T a0f992f08fb841a5b18b14f5ad0b9c58 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:52.779517 2698 raft_consensus.cc:2258] T a0f992f08fb841a5b18b14f5ad0b9c58 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:52.780579 2698 ts_tablet_manager.cc:1875] T a0f992f08fb841a5b18b14f5ad0b9c58 P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:52.789009 2698 ts_tablet_manager.cc:1888] T a0f992f08fb841a5b18b14f5ad0b9c58 P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:52.789207 2698 log.cc:1196] T a0f992f08fb841a5b18b14f5ad0b9c58 P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/a0f992f08fb841a5b18b14f5ad0b9c58
I20240501 08:12:52.794131 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet a0f992f08fb841a5b18b14f5ad0b9c58 (table rebalance_test_table_9 [id=e6efea2767a741beae5c93339e7712ba]) successfully deleted
I20240501 08:12:52.815798 30803 consensus_queue.cc:237] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
W20240501 08:12:52.819523 30737 consensus_peers.cc:485] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 -> Peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Couldn't send request to peer 91d11eae5ef6432ca86eb8ed446b6ea6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a1fde6b765374df9bc4674a5d678c920. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:52.819640 28022 raft_consensus.cc:1261] T a1fde6b765374df9bc4674a5d678c920 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:52.820070 27892 raft_consensus.cc:1261] T a1fde6b765374df9bc4674a5d678c920 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:52.820266 2227 consensus_queue.cc:1035] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:52.820811 2227 consensus_queue.cc:1035] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:52.823319 2226 raft_consensus.cc:2932] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:12:52.823804 28018 raft_consensus.cc:2932] T a1fde6b765374df9bc4674a5d678c920 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:12:52.828883 27892 raft_consensus.cc:2932] T a1fde6b765374df9bc4674a5d678c920 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:12:52.832840 27620 catalog_manager.cc:5484] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index -1 to 2, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:52.855028 30803 consensus_queue.cc:237] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:12:52.857708 30934 raft_consensus.cc:1261] T 00205a461b704a19bbd39bc71361e529 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:12:52.857766 30739 consensus_peers.cc:485] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00205a461b704a19bbd39bc71361e529. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:52.858273 30672 raft_consensus.cc:1261] T 00205a461b704a19bbd39bc71361e529 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:52.858997 2227 consensus_queue.cc:1035] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:52.859297 2216 consensus_queue.cc:1035] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:52.859757 2714 ts_tablet_manager.cc:902] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43: Initiating tablet copy from peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:52.862902 2714 tablet_copy_client.cc:308] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.136:46845
I20240501 08:12:52.863669 30957 tablet_copy_service.cc:140] P a29cd590999b4f32849ff5689cd53c11: Received BeginTabletCopySession request for tablet f63215605ade41a5bc8d97a35ce33a31 from peer 6331f77dae9d452184e8cd0cbf5f1b43 ({username='slave'} at 127.21.84.129:48739)
I20240501 08:12:52.863433 2226 raft_consensus.cc:2932] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:52.863873 30957 tablet_copy_service.cc:161] P a29cd590999b4f32849ff5689cd53c11: Beginning new tablet copy session on tablet f63215605ade41a5bc8d97a35ce33a31 from peer 6331f77dae9d452184e8cd0cbf5f1b43 at {username='slave'} at 127.21.84.129:48739: session id = 6331f77dae9d452184e8cd0cbf5f1b43-f63215605ade41a5bc8d97a35ce33a31
I20240501 08:12:52.863996 30672 raft_consensus.cc:2932] T 00205a461b704a19bbd39bc71361e529 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:52.868288 30957 tablet_copy_source_session.cc:205] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:52.871858 2714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63215605ade41a5bc8d97a35ce33a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:52.874126 27611 catalog_manager.cc:5484] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:52.877501 2714 tablet_copy_client.cc:783] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:52.877810 2714 tablet_copy_client.cc:647] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:52.879581 2714 tablet_copy_client.cc:523] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:52.885540 2714 tablet_bootstrap.cc:492] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:52.885406 30936 raft_consensus.cc:2932] T 00205a461b704a19bbd39bc71361e529 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:52.907021 2714 tablet_bootstrap.cc:492] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:52.907718 2714 tablet_bootstrap.cc:492] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap complete.
I20240501 08:12:52.908095 2714 ts_tablet_manager.cc:1367] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.000s
I20240501 08:12:52.909487 2714 raft_consensus.cc:348] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:12:52.909905 2714 raft_consensus.cc:729] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: LEARNER
I20240501 08:12:52.910189 2714 consensus_queue.cc:260] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:12:52.910856 2714 ts_tablet_manager.cc:1398] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 08:12:52.914521 30957 tablet_copy_service.cc:342] P a29cd590999b4f32849ff5689cd53c11: Request end of tablet copy session 6331f77dae9d452184e8cd0cbf5f1b43-f63215605ade41a5bc8d97a35ce33a31 received from {username='slave'} at 127.21.84.129:48739
I20240501 08:12:52.914674 30957 tablet_copy_service.cc:433] P a29cd590999b4f32849ff5689cd53c11: ending tablet copy session 6331f77dae9d452184e8cd0cbf5f1b43-f63215605ade41a5bc8d97a35ce33a31 on tablet f63215605ade41a5bc8d97a35ce33a31 with peer 6331f77dae9d452184e8cd0cbf5f1b43
I20240501 08:12:53.035288 2422 raft_consensus.cc:1050] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6: attempting to promote NON_VOTER f1525cbc82a24d13b65f136695c903af to VOTER
I20240501 08:12:53.036123 2422 consensus_queue.cc:237] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:53.038645 27892 raft_consensus.cc:1261] T 7affd631d1e444eaaf8a4c51267a204f P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:53.039058 30672 raft_consensus.cc:1261] T 7affd631d1e444eaaf8a4c51267a204f P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:53.039680 30936 raft_consensus.cc:1261] T 7affd631d1e444eaaf8a4c51267a204f P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:53.039702 2762 consensus_queue.cc:1035] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:53.040381 2594 consensus_queue.cc:1035] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:53.044350 2762 consensus_queue.cc:1035] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:53.045375 2422 raft_consensus.cc:2932] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:53.047888 30672 raft_consensus.cc:2932] T 7affd631d1e444eaaf8a4c51267a204f P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:53.060046 27620 catalog_manager.cc:5484] T 7affd631d1e444eaaf8a4c51267a204f P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:53.060235 30934 raft_consensus.cc:2932] T 7affd631d1e444eaaf8a4c51267a204f P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:53.078791 27892 raft_consensus.cc:2932] T 7affd631d1e444eaaf8a4c51267a204f P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:53.086484 30533 consensus_queue.cc:237] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:53.088935 30936 raft_consensus.cc:1261] T 7affd631d1e444eaaf8a4c51267a204f P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:53.089983 27892 raft_consensus.cc:1261] T 7affd631d1e444eaaf8a4c51267a204f P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:53.090634 2594 consensus_queue.cc:1035] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:53.096787 2772 ts_tablet_manager.cc:902] T 87bd346903dc4c2c85ecaf362fe57799 P f1525cbc82a24d13b65f136695c903af: Initiating tablet copy from peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:53.099639 2594 consensus_queue.cc:1035] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:53.102820 2422 raft_consensus.cc:2932] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:53.115607 30936 raft_consensus.cc:2932] T 7affd631d1e444eaaf8a4c51267a204f P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:53.148821 2620 raft_consensus.cc:1050] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af: attempting to promote NON_VOTER a29cd590999b4f32849ff5689cd53c11 to VOTER
I20240501 08:12:53.150431 2620 consensus_queue.cc:237] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:12:53.152130 2778 ts_tablet_manager.cc:902] T 6a5f99276dad456ea80e3c9f9618059a P aad81047009a4ffcbc450b60030583b5: Initiating tablet copy from peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:53.173771 2772 tablet_copy_client.cc:308] T 87bd346903dc4c2c85ecaf362fe57799 P f1525cbc82a24d13b65f136695c903af: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.131:36305
I20240501 08:12:53.174559 28042 tablet_copy_service.cc:140] P aad81047009a4ffcbc450b60030583b5: Received BeginTabletCopySession request for tablet 87bd346903dc4c2c85ecaf362fe57799 from peer f1525cbc82a24d13b65f136695c903af ({username='slave'} at 127.21.84.130:59663)
I20240501 08:12:53.174737 28042 tablet_copy_service.cc:161] P aad81047009a4ffcbc450b60030583b5: Beginning new tablet copy session on tablet 87bd346903dc4c2c85ecaf362fe57799 from peer f1525cbc82a24d13b65f136695c903af at {username='slave'} at 127.21.84.130:59663: session id = f1525cbc82a24d13b65f136695c903af-87bd346903dc4c2c85ecaf362fe57799
I20240501 08:12:53.177125 27611 catalog_manager.cc:5484] T 7affd631d1e444eaaf8a4c51267a204f P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:53.177673 28042 tablet_copy_source_session.cc:205] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:53.179831 2772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87bd346903dc4c2c85ecaf362fe57799. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:53.184645 30650 tablet_service.cc:1514] Processing DeleteTablet for tablet 7affd631d1e444eaaf8a4c51267a204f with delete_type TABLET_DATA_TOMBSTONED (TS 9a022e1a501f4d44a0468054d6c3de47 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57156
I20240501 08:12:53.186491 2783 tablet_replica.cc:330] T 7affd631d1e444eaaf8a4c51267a204f P 9a022e1a501f4d44a0468054d6c3de47: stopping tablet replica
I20240501 08:12:53.186877 2783 raft_consensus.cc:2229] T 7affd631d1e444eaaf8a4c51267a204f P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:53.188387 2783 raft_consensus.cc:2258] T 7affd631d1e444eaaf8a4c51267a204f P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:53.189801 2783 ts_tablet_manager.cc:1875] T 7affd631d1e444eaaf8a4c51267a204f P 9a022e1a501f4d44a0468054d6c3de47: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:53.192137 27607 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7affd631d1e444eaaf8a4c51267a204f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:53.195721 2772 tablet_copy_client.cc:783] T 87bd346903dc4c2c85ecaf362fe57799 P f1525cbc82a24d13b65f136695c903af: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:53.196002 2772 tablet_copy_client.cc:647] T 87bd346903dc4c2c85ecaf362fe57799 P f1525cbc82a24d13b65f136695c903af: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:53.197971 2772 tablet_copy_client.cc:523] T 87bd346903dc4c2c85ecaf362fe57799 P f1525cbc82a24d13b65f136695c903af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:53.211427 2783 ts_tablet_manager.cc:1888] T 7affd631d1e444eaaf8a4c51267a204f P 9a022e1a501f4d44a0468054d6c3de47: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:53.213490 2783 log.cc:1196] T 7affd631d1e444eaaf8a4c51267a204f P 9a022e1a501f4d44a0468054d6c3de47: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/wals/7affd631d1e444eaaf8a4c51267a204f
I20240501 08:12:53.214690 27604 catalog_manager.cc:4830] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): tablet 7affd631d1e444eaaf8a4c51267a204f (table rebalancer_extra_table_54 [id=20ee5c0014e146438233fd009facbf2c]) successfully deleted
I20240501 08:12:53.216238 2778 tablet_copy_client.cc:308] T 6a5f99276dad456ea80e3c9f9618059a P aad81047009a4ffcbc450b60030583b5: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.129:43471
I20240501 08:12:53.216420 27892 raft_consensus.cc:2932] T 7affd631d1e444eaaf8a4c51267a204f P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:53.217846 2772 tablet_bootstrap.cc:492] T 87bd346903dc4c2c85ecaf362fe57799 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:53.218111 27781 tablet_copy_service.cc:140] P 6331f77dae9d452184e8cd0cbf5f1b43: Received BeginTabletCopySession request for tablet 6a5f99276dad456ea80e3c9f9618059a from peer aad81047009a4ffcbc450b60030583b5 ({username='slave'} at 127.21.84.131:51285)
I20240501 08:12:53.218307 27781 tablet_copy_service.cc:161] P 6331f77dae9d452184e8cd0cbf5f1b43: Beginning new tablet copy session on tablet 6a5f99276dad456ea80e3c9f9618059a from peer aad81047009a4ffcbc450b60030583b5 at {username='slave'} at 127.21.84.131:51285: session id = aad81047009a4ffcbc450b60030583b5-6a5f99276dad456ea80e3c9f9618059a
I20240501 08:12:53.222970 27781 tablet_copy_source_session.cc:205] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:53.224169 2772 tablet_bootstrap.cc:492] T 87bd346903dc4c2c85ecaf362fe57799 P f1525cbc82a24d13b65f136695c903af: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:53.224530 2772 tablet_bootstrap.cc:492] T 87bd346903dc4c2c85ecaf362fe57799 P f1525cbc82a24d13b65f136695c903af: Bootstrap complete.
I20240501 08:12:53.224814 2772 ts_tablet_manager.cc:1367] T 87bd346903dc4c2c85ecaf362fe57799 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20240501 08:12:53.225270 2778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a5f99276dad456ea80e3c9f9618059a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:53.225493 2772 raft_consensus.cc:348] T 87bd346903dc4c2c85ecaf362fe57799 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:53.225718 2772 raft_consensus.cc:729] T 87bd346903dc4c2c85ecaf362fe57799 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: LEARNER
I20240501 08:12:53.225962 2772 consensus_queue.cc:260] T 87bd346903dc4c2c85ecaf362fe57799 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:53.226523 2772 ts_tablet_manager.cc:1398] T 87bd346903dc4c2c85ecaf362fe57799 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:53.229182 28042 tablet_copy_service.cc:342] P aad81047009a4ffcbc450b60030583b5: Request end of tablet copy session f1525cbc82a24d13b65f136695c903af-87bd346903dc4c2c85ecaf362fe57799 received from {username='slave'} at 127.21.84.130:59663
I20240501 08:12:53.229300 28042 tablet_copy_service.cc:433] P aad81047009a4ffcbc450b60030583b5: ending tablet copy session f1525cbc82a24d13b65f136695c903af-87bd346903dc4c2c85ecaf362fe57799 on tablet 87bd346903dc4c2c85ecaf362fe57799 with peer f1525cbc82a24d13b65f136695c903af
I20240501 08:12:53.241879 28149 raft_consensus.cc:1261] T 367847f920184dd6abcdcd7564f76376 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:53.242295 30936 raft_consensus.cc:1261] T 367847f920184dd6abcdcd7564f76376 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:53.243307 28018 raft_consensus.cc:1261] T 367847f920184dd6abcdcd7564f76376 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:53.244354 2620 consensus_queue.cc:1035] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:53.245900 2620 consensus_queue.cc:1035] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:53.246237 2531 consensus_queue.cc:1035] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:53.254422 2785 raft_consensus.cc:2932] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:53.266381 2778 tablet_copy_client.cc:783] T 6a5f99276dad456ea80e3c9f9618059a P aad81047009a4ffcbc450b60030583b5: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:53.266727 2778 tablet_copy_client.cc:647] T 6a5f99276dad456ea80e3c9f9618059a P aad81047009a4ffcbc450b60030583b5: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:53.267984 28022 raft_consensus.cc:2932] T 367847f920184dd6abcdcd7564f76376 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:53.269032 2778 tablet_copy_client.cc:523] T 6a5f99276dad456ea80e3c9f9618059a P aad81047009a4ffcbc450b60030583b5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:53.270735 30936 raft_consensus.cc:2932] T 367847f920184dd6abcdcd7564f76376 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:53.272915 2778 tablet_bootstrap.cc:492] T 6a5f99276dad456ea80e3c9f9618059a P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:53.279356 2778 tablet_bootstrap.cc:492] T 6a5f99276dad456ea80e3c9f9618059a P aad81047009a4ffcbc450b60030583b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:53.279852 2778 tablet_bootstrap.cc:492] T 6a5f99276dad456ea80e3c9f9618059a P aad81047009a4ffcbc450b60030583b5: Bootstrap complete.
I20240501 08:12:53.280174 2778 ts_tablet_manager.cc:1367] T 6a5f99276dad456ea80e3c9f9618059a P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20240501 08:12:53.280894 2778 raft_consensus.cc:348] T 6a5f99276dad456ea80e3c9f9618059a P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:12:53.280886 27620 catalog_manager.cc:5484] T 367847f920184dd6abcdcd7564f76376 P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:53.281301 2778 raft_consensus.cc:729] T 6a5f99276dad456ea80e3c9f9618059a P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: LEARNER
I20240501 08:12:53.281538 2778 consensus_queue.cc:260] T 6a5f99276dad456ea80e3c9f9618059a P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:12:53.282163 2778 ts_tablet_manager.cc:1398] T 6a5f99276dad456ea80e3c9f9618059a P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:53.284871 27781 tablet_copy_service.cc:342] P 6331f77dae9d452184e8cd0cbf5f1b43: Request end of tablet copy session aad81047009a4ffcbc450b60030583b5-6a5f99276dad456ea80e3c9f9618059a received from {username='slave'} at 127.21.84.131:51285
I20240501 08:12:53.285370 27781 tablet_copy_service.cc:433] P 6331f77dae9d452184e8cd0cbf5f1b43: ending tablet copy session aad81047009a4ffcbc450b60030583b5-6a5f99276dad456ea80e3c9f9618059a on tablet 6a5f99276dad456ea80e3c9f9618059a with peer aad81047009a4ffcbc450b60030583b5
I20240501 08:12:53.287429 28152 raft_consensus.cc:2932] T 367847f920184dd6abcdcd7564f76376 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:53.323678 27892 consensus_queue.cc:237] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:12:53.324558 28152 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a05223cb2f20478b8d29ffa065b4c67b"
dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43678
I20240501 08:12:53.324761 28152 raft_consensus.cc:595] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:12:53.326287 30936 raft_consensus.cc:1261] T 367847f920184dd6abcdcd7564f76376 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:53.327108 2785 consensus_queue.cc:1035] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:53.332022 28022 raft_consensus.cc:1261] T 367847f920184dd6abcdcd7564f76376 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:53.333187 2777 consensus_queue.cc:1035] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:53.340045 2786 raft_consensus.cc:2932] T 367847f920184dd6abcdcd7564f76376 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:53.341332 30936 raft_consensus.cc:2932] T 367847f920184dd6abcdcd7564f76376 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:53.348906 2797 ts_tablet_manager.cc:902] T a1fde6b765374df9bc4674a5d678c920 P 91d11eae5ef6432ca86eb8ed446b6ea6: Initiating tablet copy from peer 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:53.350533 28022 raft_consensus.cc:2932] T 367847f920184dd6abcdcd7564f76376 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:53.357203 27607 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 367847f920184dd6abcdcd7564f76376 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:53.360224 27760 raft_consensus.cc:1203] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:53.408099 27620 catalog_manager.cc:5484] T 367847f920184dd6abcdcd7564f76376 P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 5 to 6, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:53.411832 28131 tablet_service.cc:1514] Processing DeleteTablet for tablet 367847f920184dd6abcdcd7564f76376 with delete_type TABLET_DATA_TOMBSTONED (TS 91d11eae5ef6432ca86eb8ed446b6ea6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:53.418469 2797 tablet_copy_client.cc:308] T a1fde6b765374df9bc4674a5d678c920 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.135:33503
I20240501 08:12:53.419238 30825 tablet_copy_service.cc:140] P 64998323a1f4482d8704bf20594d13a5: Received BeginTabletCopySession request for tablet a1fde6b765374df9bc4674a5d678c920 from peer 91d11eae5ef6432ca86eb8ed446b6ea6 ({username='slave'} at 127.21.84.132:54591)
I20240501 08:12:53.419473 30825 tablet_copy_service.cc:161] P 64998323a1f4482d8704bf20594d13a5: Beginning new tablet copy session on tablet a1fde6b765374df9bc4674a5d678c920 from peer 91d11eae5ef6432ca86eb8ed446b6ea6 at {username='slave'} at 127.21.84.132:54591: session id = 91d11eae5ef6432ca86eb8ed446b6ea6-a1fde6b765374df9bc4674a5d678c920
I20240501 08:12:53.432228 2804 tablet_replica.cc:330] T 367847f920184dd6abcdcd7564f76376 P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:53.432571 2804 raft_consensus.cc:2229] T 367847f920184dd6abcdcd7564f76376 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:53.432906 2804 raft_consensus.cc:2258] T 367847f920184dd6abcdcd7564f76376 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:53.433881 2804 ts_tablet_manager.cc:1875] T 367847f920184dd6abcdcd7564f76376 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:53.443279 2804 ts_tablet_manager.cc:1888] T 367847f920184dd6abcdcd7564f76376 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:53.443627 2804 log.cc:1196] T 367847f920184dd6abcdcd7564f76376 P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/367847f920184dd6abcdcd7564f76376
I20240501 08:12:53.444566 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet 367847f920184dd6abcdcd7564f76376 (table rebalance_test_table_7 [id=1ae463b433bd4ed6bb9ab1bdc10dfdf6]) successfully deleted
I20240501 08:12:53.491904 2807 ts_tablet_manager.cc:902] T 00205a461b704a19bbd39bc71361e529 P 6331f77dae9d452184e8cd0cbf5f1b43: Initiating tablet copy from peer 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:53.500130 30825 tablet_copy_source_session.cc:205] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:53.502216 2797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1fde6b765374df9bc4674a5d678c920. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:53.512113 2797 tablet_copy_client.cc:783] T a1fde6b765374df9bc4674a5d678c920 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:53.512430 2797 tablet_copy_client.cc:647] T a1fde6b765374df9bc4674a5d678c920 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:53.514798 2807 tablet_copy_client.cc:308] T 00205a461b704a19bbd39bc71361e529 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.135:33503
I20240501 08:12:53.515914 30825 tablet_copy_service.cc:140] P 64998323a1f4482d8704bf20594d13a5: Received BeginTabletCopySession request for tablet 00205a461b704a19bbd39bc71361e529 from peer 6331f77dae9d452184e8cd0cbf5f1b43 ({username='slave'} at 127.21.84.129:58333)
I20240501 08:12:53.516027 2797 tablet_copy_client.cc:523] T a1fde6b765374df9bc4674a5d678c920 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:53.516081 30825 tablet_copy_service.cc:161] P 64998323a1f4482d8704bf20594d13a5: Beginning new tablet copy session on tablet 00205a461b704a19bbd39bc71361e529 from peer 6331f77dae9d452184e8cd0cbf5f1b43 at {username='slave'} at 127.21.84.129:58333: session id = 6331f77dae9d452184e8cd0cbf5f1b43-00205a461b704a19bbd39bc71361e529
I20240501 08:12:53.518040 30825 tablet_copy_source_session.cc:205] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:53.518704 2797 tablet_bootstrap.cc:492] T a1fde6b765374df9bc4674a5d678c920 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:12:53.519234 2807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00205a461b704a19bbd39bc71361e529. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:53.525390 2807 tablet_copy_client.cc:783] T 00205a461b704a19bbd39bc71361e529 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:53.525662 2807 tablet_copy_client.cc:647] T 00205a461b704a19bbd39bc71361e529 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:53.528992 2807 tablet_copy_client.cc:523] T 00205a461b704a19bbd39bc71361e529 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:53.532169 2807 tablet_bootstrap.cc:492] T 00205a461b704a19bbd39bc71361e529 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:53.532327 28022 raft_consensus.cc:1203] T 6a5f99276dad456ea80e3c9f9618059a P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:53.546984 2797 tablet_bootstrap.cc:492] T a1fde6b765374df9bc4674a5d678c920 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:53.547106 2807 tablet_bootstrap.cc:492] T 00205a461b704a19bbd39bc71361e529 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:53.547391 2797 tablet_bootstrap.cc:492] T a1fde6b765374df9bc4674a5d678c920 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap complete.
I20240501 08:12:53.547518 2807 tablet_bootstrap.cc:492] T 00205a461b704a19bbd39bc71361e529 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap complete.
I20240501 08:12:53.547657 2797 ts_tablet_manager.cc:1367] T a1fde6b765374df9bc4674a5d678c920 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.004s
I20240501 08:12:53.547791 2807 ts_tablet_manager.cc:1367] T 00205a461b704a19bbd39bc71361e529 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.008s
I20240501 08:12:53.548429 2797 raft_consensus.cc:348] T a1fde6b765374df9bc4674a5d678c920 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:12:53.548669 2797 raft_consensus.cc:729] T a1fde6b765374df9bc4674a5d678c920 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: LEARNER
I20240501 08:12:53.548636 2807 raft_consensus.cc:348] T 00205a461b704a19bbd39bc71361e529 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:12:53.548918 2807 raft_consensus.cc:729] T 00205a461b704a19bbd39bc71361e529 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: LEARNER
I20240501 08:12:53.548918 2797 consensus_queue.cc:260] T a1fde6b765374df9bc4674a5d678c920 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:12:53.549127 2807 consensus_queue.cc:260] T 00205a461b704a19bbd39bc71361e529 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:12:53.549551 2797 ts_tablet_manager.cc:1398] T a1fde6b765374df9bc4674a5d678c920 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:53.549674 2807 ts_tablet_manager.cc:1398] T 00205a461b704a19bbd39bc71361e529 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:53.550814 30825 tablet_copy_service.cc:342] P 64998323a1f4482d8704bf20594d13a5: Request end of tablet copy session 91d11eae5ef6432ca86eb8ed446b6ea6-a1fde6b765374df9bc4674a5d678c920 received from {username='slave'} at 127.21.84.132:54591
I20240501 08:12:53.550933 30825 tablet_copy_service.cc:433] P 64998323a1f4482d8704bf20594d13a5: ending tablet copy session 91d11eae5ef6432ca86eb8ed446b6ea6-a1fde6b765374df9bc4674a5d678c920 on tablet a1fde6b765374df9bc4674a5d678c920 with peer 91d11eae5ef6432ca86eb8ed446b6ea6
I20240501 08:12:53.552330 30825 tablet_copy_service.cc:342] P 64998323a1f4482d8704bf20594d13a5: Request end of tablet copy session 6331f77dae9d452184e8cd0cbf5f1b43-00205a461b704a19bbd39bc71361e529 received from {username='slave'} at 127.21.84.129:58333
I20240501 08:12:53.552440 30825 tablet_copy_service.cc:433] P 64998323a1f4482d8704bf20594d13a5: ending tablet copy session 6331f77dae9d452184e8cd0cbf5f1b43-00205a461b704a19bbd39bc71361e529 on tablet 00205a461b704a19bbd39bc71361e529 with peer 6331f77dae9d452184e8cd0cbf5f1b43
I20240501 08:12:53.588291 27892 raft_consensus.cc:1203] T 87bd346903dc4c2c85ecaf362fe57799 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:53.598634 2222 raft_consensus.cc:979] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6: : Instructing follower 6331f77dae9d452184e8cd0cbf5f1b43 to start an election
I20240501 08:12:53.598798 2222 raft_consensus.cc:1067] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Signalling peer 6331f77dae9d452184e8cd0cbf5f1b43 to start an election
I20240501 08:12:53.599484 27760 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a05223cb2f20478b8d29ffa065b4c67b"
dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
from {username='slave'} at 127.21.84.132:59727
I20240501 08:12:53.599709 27760 raft_consensus.cc:482] T a05223cb2f20478b8d29ffa065b4c67b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:53.599802 27760 raft_consensus.cc:3037] T a05223cb2f20478b8d29ffa065b4c67b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:53.602094 27760 raft_consensus.cc:504] T a05223cb2f20478b8d29ffa065b4c67b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:12:53.606099 28152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05223cb2f20478b8d29ffa065b4c67b" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:12:53.606057 28018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05223cb2f20478b8d29ffa065b4c67b" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:12:53.606305 28018 raft_consensus.cc:3037] T a05223cb2f20478b8d29ffa065b4c67b P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:53.606321 28152 raft_consensus.cc:3032] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:12:53.606401 28152 raft_consensus.cc:729] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:12:53.606698 28152 consensus_queue.cc:260] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:12:53.607074 28152 raft_consensus.cc:3037] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:53.609282 28018 raft_consensus.cc:2454] T a05223cb2f20478b8d29ffa065b4c67b P aad81047009a4ffcbc450b60030583b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 2.
I20240501 08:12:53.611449 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05223cb2f20478b8d29ffa065b4c67b" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:12:53.611660 30936 raft_consensus.cc:3037] T a05223cb2f20478b8d29ffa065b4c67b P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:53.613286 30800 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c5f4c78e69fb4c67ad65a7b0c1466798"
dest_uuid: "64998323a1f4482d8704bf20594d13a5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45032
I20240501 08:12:53.613490 30800 raft_consensus.cc:595] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:12:53.613934 30936 raft_consensus.cc:2454] T a05223cb2f20478b8d29ffa065b4c67b P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 2.
I20240501 08:12:53.614526 28152 raft_consensus.cc:2454] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 2.
I20240501 08:12:53.614853 27760 leader_election.cc:288] T a05223cb2f20478b8d29ffa065b4c67b P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 2 election: Requested vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:53.615828 27696 leader_election.cc:302] T a05223cb2f20478b8d29ffa065b4c67b P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, a29cd590999b4f32849ff5689cd53c11, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:12:53.616144 2670 raft_consensus.cc:2781] T a05223cb2f20478b8d29ffa065b4c67b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:12:53.616290 2670 raft_consensus.cc:686] T a05223cb2f20478b8d29ffa065b4c67b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:12:53.616581 2670 consensus_queue.cc:237] T a05223cb2f20478b8d29ffa065b4c67b P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:12:53.641302 27620 catalog_manager.cc:5484] T a05223cb2f20478b8d29ffa065b4c67b P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 1 to 2, leader changed from 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 2 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:53.686722 28022 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "73f40a7ef6954b34a48d8c5aeeaebc0a"
dest_uuid: "aad81047009a4ffcbc450b60030583b5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36886
I20240501 08:12:53.686977 28022 raft_consensus.cc:595] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:12:53.776268 27889 consensus_queue.cc:237] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
I20240501 08:12:53.778899 27760 raft_consensus.cc:1261] T 728616fd583a4afea00e8fe52880e55b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:53.779249 28152 raft_consensus.cc:1261] T 728616fd583a4afea00e8fe52880e55b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:53.779800 2786 consensus_queue.cc:1035] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:53.781212 2786 consensus_queue.cc:1035] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:12:53.781896 27826 consensus_peers.cc:485] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af -> Peer 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): Couldn't send request to peer 64998323a1f4482d8704bf20594d13a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 728616fd583a4afea00e8fe52880e55b. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:53.784375 2786 raft_consensus.cc:2932] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:12:53.788964 27760 raft_consensus.cc:2932] T 728616fd583a4afea00e8fe52880e55b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:12:53.792512 27620 catalog_manager.cc:5484] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af reported cstate change: config changed from index -1 to 4, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:53.793330 28152 raft_consensus.cc:2932] T 728616fd583a4afea00e8fe52880e55b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:12:53.821574 30936 consensus_queue.cc:237] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:53.823803 30672 raft_consensus.cc:1261] T 4e58bc2dffad423f942ce50144030da4 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:53.823978 30534 raft_consensus.cc:1261] T 4e58bc2dffad423f942ce50144030da4 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:12:53.824095 30873 consensus_peers.cc:485] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e58bc2dffad423f942ce50144030da4. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:53.824385 2755 consensus_queue.cc:1035] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:53.825089 2602 consensus_queue.cc:1035] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:53.828979 2602 raft_consensus.cc:2932] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:53.829604 30534 raft_consensus.cc:2932] T 4e58bc2dffad423f942ce50144030da4 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:53.831441 30669 raft_consensus.cc:2932] T 4e58bc2dffad423f942ce50144030da4 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:53.836696 27620 catalog_manager.cc:5484] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:53.855388 1772 raft_consensus.cc:979] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5: : Instructing follower 6331f77dae9d452184e8cd0cbf5f1b43 to start an election
I20240501 08:12:53.855574 1772 raft_consensus.cc:1067] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Signalling peer 6331f77dae9d452184e8cd0cbf5f1b43 to start an election
I20240501 08:12:53.856307 27760 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "73f40a7ef6954b34a48d8c5aeeaebc0a"
dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
from {username='slave'} at 127.21.84.131:51285
I20240501 08:12:53.856513 27760 raft_consensus.cc:482] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:53.856624 27760 raft_consensus.cc:3037] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:53.859041 27760 raft_consensus.cc:504] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:12:53.860045 27760 leader_election.cc:288] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 2 election: Requested vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:53.860366 30800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f40a7ef6954b34a48d8c5aeeaebc0a" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:53.860553 30800 raft_consensus.cc:3037] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:53.860527 30803 consensus_queue.cc:237] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:12:53.860955 30534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f40a7ef6954b34a48d8c5aeeaebc0a" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:53.860975 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f40a7ef6954b34a48d8c5aeeaebc0a" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:12:53.861150 30534 raft_consensus.cc:3037] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:53.861162 28022 raft_consensus.cc:3032] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:12:53.861248 28022 raft_consensus.cc:729] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Running, Role: LEADER
I20240501 08:12:53.861557 28022 consensus_queue.cc:260] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:12:53.862007 28022 raft_consensus.cc:3037] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:53.863698 30800 raft_consensus.cc:2454] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 2.
I20240501 08:12:53.863919 30534 raft_consensus.cc:2454] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 2.
I20240501 08:12:53.865712 27697 leader_election.cc:302] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 6331f77dae9d452184e8cd0cbf5f1b43, 64998323a1f4482d8704bf20594d13a5; no voters:
I20240501 08:12:53.867179 2674 raft_consensus.cc:2781] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:12:53.867305 2674 raft_consensus.cc:686] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:12:53.867584 2674 consensus_queue.cc:237] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:12:53.874099 28022 raft_consensus.cc:2454] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 2.
W20240501 08:12:53.877075 30739 consensus_peers.cc:485] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 -> Peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Couldn't send request to peer aad81047009a4ffcbc450b60030583b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6ad187384514267b57cc994c9612709. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:53.878446 30936 raft_consensus.cc:1261] T b6ad187384514267b57cc994c9612709 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:53.879103 2216 consensus_queue.cc:1035] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:53.881181 28152 raft_consensus.cc:1261] T b6ad187384514267b57cc994c9612709 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:53.881693 2216 consensus_queue.cc:1035] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:53.884961 27620 catalog_manager.cc:5484] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 1 to 2, leader changed from aad81047009a4ffcbc450b60030583b5 (127.21.84.131) to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 2 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:53.887530 2226 raft_consensus.cc:2932] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:53.892954 30936 raft_consensus.cc:2932] T b6ad187384514267b57cc994c9612709 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:53.901301 27620 catalog_manager.cc:5484] T b6ad187384514267b57cc994c9612709 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:53.912582 28152 raft_consensus.cc:2932] T b6ad187384514267b57cc994c9612709 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:53.918074 28149 raft_consensus.cc:1203] T a1fde6b765374df9bc4674a5d678c920 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:53.929446 30669 consensus_queue.cc:237] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:12:53.932658 30936 raft_consensus.cc:1261] T 462c892043d8475c99389c6580a66f0c P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:53.932695 30803 raft_consensus.cc:1261] T 462c892043d8475c99389c6580a66f0c P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:53.933223 1679 consensus_queue.cc:1035] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:53.933557 2436 consensus_queue.cc:1035] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:12:53.934441 30607 consensus_peers.cc:485] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 462c892043d8475c99389c6580a66f0c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:53.947290 2436 raft_consensus.cc:2932] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:53.950815 30803 raft_consensus.cc:2932] T 462c892043d8475c99389c6580a66f0c P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:53.955132 27611 catalog_manager.cc:5484] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New cstate: current_term: 1 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:53.967475 2226 raft_consensus.cc:979] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5: : Instructing follower 91d11eae5ef6432ca86eb8ed446b6ea6 to start an election
I20240501 08:12:53.967638 2226 raft_consensus.cc:1067] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Signalling peer 91d11eae5ef6432ca86eb8ed446b6ea6 to start an election
I20240501 08:12:53.968353 28152 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c5f4c78e69fb4c67ad65a7b0c1466798"
dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
from {username='slave'} at 127.21.84.135:48917
I20240501 08:12:53.968608 28152 raft_consensus.cc:482] T c5f4c78e69fb4c67ad65a7b0c1466798 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:53.968714 28152 raft_consensus.cc:3037] T c5f4c78e69fb4c67ad65a7b0c1466798 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:53.970635 30936 raft_consensus.cc:2932] T 462c892043d8475c99389c6580a66f0c P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:53.971371 28152 raft_consensus.cc:504] T c5f4c78e69fb4c67ad65a7b0c1466798 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:12:53.972401 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5f4c78e69fb4c67ad65a7b0c1466798" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:53.972741 30803 raft_consensus.cc:3032] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:12:53.972697 28022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5f4c78e69fb4c67ad65a7b0c1466798" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "aad81047009a4ffcbc450b60030583b5"
I20240501 08:12:53.972909 28022 raft_consensus.cc:3037] T c5f4c78e69fb4c67ad65a7b0c1466798 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:53.972890 30803 raft_consensus.cc:729] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Running, Role: LEADER
I20240501 08:12:53.973809 30803 consensus_queue.cc:260] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:12:53.974309 30803 raft_consensus.cc:3037] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:53.975607 28022 raft_consensus.cc:2454] T c5f4c78e69fb4c67ad65a7b0c1466798 P aad81047009a4ffcbc450b60030583b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 2.
I20240501 08:12:53.976845 28152 leader_election.cc:288] T c5f4c78e69fb4c67ad65a7b0c1466798 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 2 election: Requested vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:53.976948 30534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5f4c78e69fb4c67ad65a7b0c1466798" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:12:53.977180 30534 raft_consensus.cc:3037] T c5f4c78e69fb4c67ad65a7b0c1466798 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:53.981158 30803 raft_consensus.cc:2454] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 2.
I20240501 08:12:53.981537 28086 leader_election.cc:302] T c5f4c78e69fb4c67ad65a7b0c1466798 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 91d11eae5ef6432ca86eb8ed446b6ea6, aad81047009a4ffcbc450b60030583b5; no voters:
I20240501 08:12:53.981786 2607 raft_consensus.cc:2781] T c5f4c78e69fb4c67ad65a7b0c1466798 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:12:53.981896 2607 raft_consensus.cc:686] T c5f4c78e69fb4c67ad65a7b0c1466798 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 LEADER]: Becoming Leader. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Running, Role: LEADER
I20240501 08:12:53.982225 30534 raft_consensus.cc:2454] T c5f4c78e69fb4c67ad65a7b0c1466798 P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 2.
I20240501 08:12:53.982198 2607 consensus_queue.cc:237] T c5f4c78e69fb4c67ad65a7b0c1466798 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:12:53.985885 2755 raft_consensus.cc:1050] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11: attempting to promote NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 to VOTER
I20240501 08:12:53.986908 2755 consensus_queue.cc:237] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:12:53.989647 27760 raft_consensus.cc:1203] T 00205a461b704a19bbd39bc71361e529 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:53.997740 28022 raft_consensus.cc:1261] T f63215605ade41a5bc8d97a35ce33a31 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:53.998119 30534 raft_consensus.cc:1261] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:53.998682 2755 consensus_queue.cc:1035] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:53.999020 2894 consensus_queue.cc:1035] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:53.999433 27620 catalog_manager.cc:5484] T c5f4c78e69fb4c67ad65a7b0c1466798 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: term changed from 1 to 2, leader changed from 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) to 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132). New cstate: current_term: 2 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:54.003262 27758 raft_consensus.cc:1261] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:54.004323 2755 consensus_queue.cc:1035] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 08:12:54.012391 2894 raft_consensus.cc:2932] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:54.017235 27758 raft_consensus.cc:2932] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:54.020892 28022 raft_consensus.cc:2932] T f63215605ade41a5bc8d97a35ce33a31 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:54.030722 27620 catalog_manager.cc:5484] T f63215605ade41a5bc8d97a35ce33a31 P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:54.032788 30534 raft_consensus.cc:2932] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:54.041644 30936 consensus_queue.cc:237] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:12:54.046442 27758 raft_consensus.cc:1261] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:54.047259 30534 raft_consensus.cc:1261] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:54.048166 2894 consensus_queue.cc:1035] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:54.048684 2902 raft_consensus.cc:1050] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43: attempting to promote NON_VOTER aad81047009a4ffcbc450b60030583b5 to VOTER
I20240501 08:12:54.049513 2902 consensus_queue.cc:237] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } }
I20240501 08:12:54.049779 2894 consensus_queue.cc:1035] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:54.050711 2602 raft_consensus.cc:2932] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:54.051352 30533 raft_consensus.cc:2932] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:54.055544 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f63215605ade41a5bc8d97a35ce33a31 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:54.059163 27620 catalog_manager.cc:5484] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index 3 to 4, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:54.062801 27889 raft_consensus.cc:1261] T 6a5f99276dad456ea80e3c9f9618059a P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:54.063769 27758 raft_consensus.cc:2932] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:54.064211 2670 consensus_queue.cc:1035] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:54.065052 28022 raft_consensus.cc:1261] T 6a5f99276dad456ea80e3c9f9618059a P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:54.068065 2670 consensus_queue.cc:1035] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:54.070569 28000 tablet_service.cc:1514] Processing DeleteTablet for tablet f63215605ade41a5bc8d97a35ce33a31 with delete_type TABLET_DATA_TOMBSTONED (TS aad81047009a4ffcbc450b60030583b5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:54.072916 2674 raft_consensus.cc:2932] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:54.074257 30803 raft_consensus.cc:1261] T 6a5f99276dad456ea80e3c9f9618059a P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:54.077315 2670 consensus_queue.cc:1035] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:54.081589 28022 raft_consensus.cc:2932] T 6a5f99276dad456ea80e3c9f9618059a P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:54.081727 30803 raft_consensus.cc:2932] T 6a5f99276dad456ea80e3c9f9618059a P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:54.086565 2936 tablet_replica.cc:330] T f63215605ade41a5bc8d97a35ce33a31 P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:54.086800 2936 raft_consensus.cc:2229] T f63215605ade41a5bc8d97a35ce33a31 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:54.087018 2936 raft_consensus.cc:2258] T f63215605ade41a5bc8d97a35ce33a31 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:54.089104 2936 ts_tablet_manager.cc:1875] T f63215605ade41a5bc8d97a35ce33a31 P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:54.089251 27620 catalog_manager.cc:5484] T 6a5f99276dad456ea80e3c9f9618059a P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:54.094223 2832 raft_consensus.cc:1050] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5: attempting to promote NON_VOTER f1525cbc82a24d13b65f136695c903af to VOTER
I20240501 08:12:54.095088 2832 consensus_queue.cc:237] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:54.098688 30534 raft_consensus.cc:1261] T 87bd346903dc4c2c85ecaf362fe57799 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:54.098677 27889 raft_consensus.cc:1261] T 87bd346903dc4c2c85ecaf362fe57799 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:54.098973 30672 raft_consensus.cc:1261] T 87bd346903dc4c2c85ecaf362fe57799 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:54.099635 1772 consensus_queue.cc:1035] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:54.100359 1772 consensus_queue.cc:1035] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:54.100941 1772 consensus_queue.cc:1035] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:54.123402 2444 raft_consensus.cc:2932] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:54.127636 30670 raft_consensus.cc:2932] T 87bd346903dc4c2c85ecaf362fe57799 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:54.132267 27892 raft_consensus.cc:2932] T 6a5f99276dad456ea80e3c9f9618059a P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:54.137027 27612 catalog_manager.cc:5484] T 87bd346903dc4c2c85ecaf362fe57799 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:54.142006 27758 consensus_queue.cc:237] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } }
I20240501 08:12:54.144006 27892 raft_consensus.cc:1261] T 6a5f99276dad456ea80e3c9f9618059a P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:54.144507 2670 consensus_queue.cc:1035] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:54.148095 2670 raft_consensus.cc:2932] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:54.152029 27891 raft_consensus.cc:2932] T 6a5f99276dad456ea80e3c9f9618059a P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:54.153776 2936 ts_tablet_manager.cc:1888] T f63215605ade41a5bc8d97a35ce33a31 P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:54.155917 2936 log.cc:1196] T f63215605ade41a5bc8d97a35ce33a31 P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/f63215605ade41a5bc8d97a35ce33a31
I20240501 08:12:54.156708 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet f63215605ade41a5bc8d97a35ce33a31 (table rebalancer_extra_table_44 [id=a727b21f159445ba8749f9bf1bec2b3d]) successfully deleted
I20240501 08:12:54.156708 30533 raft_consensus.cc:2932] T 87bd346903dc4c2c85ecaf362fe57799 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:54.160646 28022 raft_consensus.cc:1261] T 6a5f99276dad456ea80e3c9f9618059a P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:54.163040 2933 consensus_queue.cc:1035] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:54.163552 27889 raft_consensus.cc:2932] T 87bd346903dc4c2c85ecaf362fe57799 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:54.165791 28022 raft_consensus.cc:2932] T 6a5f99276dad456ea80e3c9f9618059a P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:54.167198 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6a5f99276dad456ea80e3c9f9618059a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:54.175459 30936 raft_consensus.cc:1261] T a05223cb2f20478b8d29ffa065b4c67b P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:12:54.177076 2902 consensus_queue.cc:1035] T a05223cb2f20478b8d29ffa065b4c67b P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:54.178244 27619 catalog_manager.cc:5484] T 6a5f99276dad456ea80e3c9f9618059a P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index 3 to 4, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:54.182475 28018 consensus_queue.cc:237] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:54.183670 28016 raft_consensus.cc:1261] T a05223cb2f20478b8d29ffa065b4c67b P aad81047009a4ffcbc450b60030583b5 [term 2 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:12:54.185796 30534 raft_consensus.cc:1261] T 87bd346903dc4c2c85ecaf362fe57799 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:54.185992 30784 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a5f99276dad456ea80e3c9f9618059a with delete_type TABLET_DATA_TOMBSTONED (TS 64998323a1f4482d8704bf20594d13a5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38952
I20240501 08:12:54.188093 2933 consensus_queue.cc:1035] T a05223cb2f20478b8d29ffa065b4c67b P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:54.188220 2832 consensus_queue.cc:1035] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:54.190783 2444 raft_consensus.cc:2932] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:54.196525 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 87bd346903dc4c2c85ecaf362fe57799 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:54.199635 30534 raft_consensus.cc:2932] T 87bd346903dc4c2c85ecaf362fe57799 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:54.203595 27889 raft_consensus.cc:1261] T 87bd346903dc4c2c85ecaf362fe57799 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:54.205772 2952 tablet_replica.cc:330] T 6a5f99276dad456ea80e3c9f9618059a P 64998323a1f4482d8704bf20594d13a5: stopping tablet replica
I20240501 08:12:54.206111 2952 raft_consensus.cc:2229] T 6a5f99276dad456ea80e3c9f9618059a P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:54.206521 2952 raft_consensus.cc:2258] T 6a5f99276dad456ea80e3c9f9618059a P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:54.207901 2952 ts_tablet_manager.cc:1875] T 6a5f99276dad456ea80e3c9f9618059a P 64998323a1f4482d8704bf20594d13a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:54.209491 2444 consensus_queue.cc:1035] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:54.210160 27619 catalog_manager.cc:5484] T 87bd346903dc4c2c85ecaf362fe57799 P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:54.215850 30650 tablet_service.cc:1514] Processing DeleteTablet for tablet 87bd346903dc4c2c85ecaf362fe57799 with delete_type TABLET_DATA_TOMBSTONED (TS 9a022e1a501f4d44a0468054d6c3de47 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57156
I20240501 08:12:54.218829 27889 raft_consensus.cc:2932] T 87bd346903dc4c2c85ecaf362fe57799 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:54.241080 28149 raft_consensus.cc:1261] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:12:54.241806 2933 consensus_queue.cc:1035] T a05223cb2f20478b8d29ffa065b4c67b P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:54.253644 2956 tablet_replica.cc:330] T 87bd346903dc4c2c85ecaf362fe57799 P 9a022e1a501f4d44a0468054d6c3de47: stopping tablet replica
I20240501 08:12:54.254776 2956 raft_consensus.cc:2229] T 87bd346903dc4c2c85ecaf362fe57799 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:54.262445 2956 raft_consensus.cc:2258] T 87bd346903dc4c2c85ecaf362fe57799 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:54.262938 2952 ts_tablet_manager.cc:1888] T 6a5f99276dad456ea80e3c9f9618059a P 64998323a1f4482d8704bf20594d13a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:54.263177 2952 log.cc:1196] T 6a5f99276dad456ea80e3c9f9618059a P 64998323a1f4482d8704bf20594d13a5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal/wals/6a5f99276dad456ea80e3c9f9618059a
I20240501 08:12:54.266799 27605 catalog_manager.cc:4830] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): tablet 6a5f99276dad456ea80e3c9f9618059a (table rebalancer_extra_table_14 [id=bd2ed215ec044ce782fe6f7c802797e6]) successfully deleted
I20240501 08:12:54.266858 2956 ts_tablet_manager.cc:1875] T 87bd346903dc4c2c85ecaf362fe57799 P 9a022e1a501f4d44a0468054d6c3de47: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:54.267251 27758 consensus_queue.cc:237] T a05223cb2f20478b8d29ffa065b4c67b P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:12:54.269974 30936 raft_consensus.cc:1261] T a05223cb2f20478b8d29ffa065b4c67b P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:12:54.270766 2933 consensus_queue.cc:1035] T a05223cb2f20478b8d29ffa065b4c67b P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:54.273597 2933 raft_consensus.cc:2932] T a05223cb2f20478b8d29ffa065b4c67b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:54.280998 30936 raft_consensus.cc:2932] T a05223cb2f20478b8d29ffa065b4c67b P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:54.286348 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a05223cb2f20478b8d29ffa065b4c67b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:54.287674 2956 ts_tablet_manager.cc:1888] T 87bd346903dc4c2c85ecaf362fe57799 P 9a022e1a501f4d44a0468054d6c3de47: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:54.290331 2956 log.cc:1196] T 87bd346903dc4c2c85ecaf362fe57799 P 9a022e1a501f4d44a0468054d6c3de47: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/wals/87bd346903dc4c2c85ecaf362fe57799
I20240501 08:12:54.291312 27604 catalog_manager.cc:4830] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): tablet 87bd346903dc4c2c85ecaf362fe57799 (table rebalancer_extra_table_42 [id=6aa96b40a8e74b81bfc5ccab950a210d]) successfully deleted
I20240501 08:12:54.291298 27611 catalog_manager.cc:5484] T a05223cb2f20478b8d29ffa065b4c67b P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index 5 to 7, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New cstate: current_term: 2 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:54.292835 28022 raft_consensus.cc:1261] T a05223cb2f20478b8d29ffa065b4c67b P aad81047009a4ffcbc450b60030583b5 [term 2 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:12:54.294186 2674 consensus_queue.cc:1035] T a05223cb2f20478b8d29ffa065b4c67b P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:54.296273 28131 tablet_service.cc:1514] Processing DeleteTablet for tablet a05223cb2f20478b8d29ffa065b4c67b with delete_type TABLET_DATA_TOMBSTONED (TS 91d11eae5ef6432ca86eb8ed446b6ea6 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:55038
I20240501 08:12:54.296326 28016 raft_consensus.cc:2932] T a05223cb2f20478b8d29ffa065b4c67b P aad81047009a4ffcbc450b60030583b5 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:54.308707 2963 tablet_replica.cc:330] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:12:54.309041 2963 raft_consensus.cc:2229] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:54.309247 2963 raft_consensus.cc:2258] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:54.310672 2963 ts_tablet_manager.cc:1875] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:54.311738 30803 raft_consensus.cc:1261] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:12:54.312243 2933 consensus_queue.cc:1035] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:54.321462 2963 ts_tablet_manager.cc:1888] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 08:12:54.321666 2963 log.cc:1196] T a05223cb2f20478b8d29ffa065b4c67b P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/a05223cb2f20478b8d29ffa065b4c67b
I20240501 08:12:54.323326 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet a05223cb2f20478b8d29ffa065b4c67b (table rebalance_test_table_15 [id=d3eb6ca7d8cf4bfdb114edb23abb2ed2]) successfully deleted
I20240501 08:12:54.336010 2906 raft_consensus.cc:1050] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5: attempting to promote NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 to VOTER
I20240501 08:12:54.337188 2906 consensus_queue.cc:237] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:12:54.355274 2967 ts_tablet_manager.cc:902] T 4e58bc2dffad423f942ce50144030da4 P f1525cbc82a24d13b65f136695c903af: Initiating tablet copy from peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:12:54.365190 28149 raft_consensus.cc:1261] T a1fde6b765374df9bc4674a5d678c920 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:54.366096 27891 raft_consensus.cc:1261] T a1fde6b765374df9bc4674a5d678c920 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:54.366657 2907 consensus_queue.cc:1035] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:54.367256 2907 consensus_queue.cc:1035] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:54.368077 28016 raft_consensus.cc:1261] T a1fde6b765374df9bc4674a5d678c920 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:54.371944 2907 consensus_queue.cc:1035] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:54.383998 2216 raft_consensus.cc:2932] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:54.388285 27891 raft_consensus.cc:2932] T a1fde6b765374df9bc4674a5d678c920 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:54.389266 2977 ts_tablet_manager.cc:902] T 462c892043d8475c99389c6580a66f0c P 6331f77dae9d452184e8cd0cbf5f1b43: Initiating tablet copy from peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:12:54.392208 28149 raft_consensus.cc:2932] T a1fde6b765374df9bc4674a5d678c920 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:54.397361 27615 catalog_manager.cc:5484] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:54.400887 2976 ts_tablet_manager.cc:902] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5: Initiating tablet copy from peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:12:54.414904 28022 raft_consensus.cc:2932] T a1fde6b765374df9bc4674a5d678c920 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:54.427105 30533 raft_consensus.cc:1261] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:12:54.428306 2902 consensus_queue.cc:1035] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:54.435011 28016 raft_consensus.cc:1261] T c5f4c78e69fb4c67ad65a7b0c1466798 P aad81047009a4ffcbc450b60030583b5 [term 2 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:12:54.435619 2222 consensus_queue.cc:1035] T c5f4c78e69fb4c67ad65a7b0c1466798 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:54.436393 2906 raft_consensus.cc:1050] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5: attempting to promote NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 to VOTER
I20240501 08:12:54.438382 2906 consensus_queue.cc:237] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:12:54.445415 30803 consensus_queue.cc:237] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:12:54.447939 28152 raft_consensus.cc:1261] T a1fde6b765374df9bc4674a5d678c920 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:54.448491 28020 raft_consensus.cc:1261] T a1fde6b765374df9bc4674a5d678c920 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:54.448603 2216 consensus_queue.cc:1035] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:54.449154 2216 consensus_queue.cc:1035] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:54.452720 2967 tablet_copy_client.cc:308] T 4e58bc2dffad423f942ce50144030da4 P f1525cbc82a24d13b65f136695c903af: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.136:46845
I20240501 08:12:54.453580 2216 raft_consensus.cc:2932] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:54.455643 28016 raft_consensus.cc:2932] T a1fde6b765374df9bc4674a5d678c920 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:54.457701 30957 tablet_copy_service.cc:140] P a29cd590999b4f32849ff5689cd53c11: Received BeginTabletCopySession request for tablet 4e58bc2dffad423f942ce50144030da4 from peer f1525cbc82a24d13b65f136695c903af ({username='slave'} at 127.21.84.130:49339)
I20240501 08:12:54.457880 30957 tablet_copy_service.cc:161] P a29cd590999b4f32849ff5689cd53c11: Beginning new tablet copy session on tablet 4e58bc2dffad423f942ce50144030da4 from peer f1525cbc82a24d13b65f136695c903af at {username='slave'} at 127.21.84.130:49339: session id = f1525cbc82a24d13b65f136695c903af-4e58bc2dffad423f942ce50144030da4
I20240501 08:12:54.463691 27612 catalog_manager.cc:5484] T a1fde6b765374df9bc4674a5d678c920 P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 3 to 4, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:54.464061 28020 raft_consensus.cc:1261] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [term 2 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:12:54.465562 28152 raft_consensus.cc:2932] T a1fde6b765374df9bc4674a5d678c920 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:54.466665 30957 tablet_copy_source_session.cc:205] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:54.466859 2933 consensus_queue.cc:1035] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:54.470490 27605 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a1fde6b765374df9bc4674a5d678c920 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:54.470583 27871 tablet_service.cc:1514] Processing DeleteTablet for tablet a1fde6b765374df9bc4674a5d678c920 with delete_type TABLET_DATA_TOMBSTONED (TS f1525cbc82a24d13b65f136695c903af not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:54.470814 2967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e58bc2dffad423f942ce50144030da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:54.476167 30803 raft_consensus.cc:1261] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:12:54.477454 2222 consensus_queue.cc:1035] T c5f4c78e69fb4c67ad65a7b0c1466798 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:54.482069 2984 ts_tablet_manager.cc:902] T b6ad187384514267b57cc994c9612709 P aad81047009a4ffcbc450b60030583b5: Initiating tablet copy from peer 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:54.488129 30936 raft_consensus.cc:1261] T 00205a461b704a19bbd39bc71361e529 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:54.487390 27758 raft_consensus.cc:1261] T 00205a461b704a19bbd39bc71361e529 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:54.490593 30672 raft_consensus.cc:1261] T 00205a461b704a19bbd39bc71361e529 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:54.493182 30534 raft_consensus.cc:1261] T c5f4c78e69fb4c67ad65a7b0c1466798 P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:12:54.493467 2226 consensus_queue.cc:1035] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:54.493829 2222 consensus_queue.cc:1035] T c5f4c78e69fb4c67ad65a7b0c1466798 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:54.494091 2226 consensus_queue.cc:1035] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:54.494935 2905 consensus_queue.cc:1035] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:54.495023 2967 tablet_copy_client.cc:783] T 4e58bc2dffad423f942ce50144030da4 P f1525cbc82a24d13b65f136695c903af: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:54.495355 2967 tablet_copy_client.cc:647] T 4e58bc2dffad423f942ce50144030da4 P f1525cbc82a24d13b65f136695c903af: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:54.501459 2906 raft_consensus.cc:2932] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:54.502825 30672 raft_consensus.cc:2932] T 00205a461b704a19bbd39bc71361e529 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:54.503722 30936 raft_consensus.cc:2932] T 00205a461b704a19bbd39bc71361e529 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:54.509478 27758 raft_consensus.cc:2932] T 00205a461b704a19bbd39bc71361e529 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:54.510883 2967 tablet_copy_client.cc:523] T 4e58bc2dffad423f942ce50144030da4 P f1525cbc82a24d13b65f136695c903af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:54.514369 2967 tablet_bootstrap.cc:492] T 4e58bc2dffad423f942ce50144030da4 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:54.516335 27611 catalog_manager.cc:5484] T 00205a461b704a19bbd39bc71361e529 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:54.529868 30802 consensus_queue.cc:237] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:12:54.533545 27758 raft_consensus.cc:1261] T 00205a461b704a19bbd39bc71361e529 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:54.534490 2226 consensus_queue.cc:1035] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:54.540994 30670 raft_consensus.cc:1261] T 00205a461b704a19bbd39bc71361e529 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:54.543263 2226 consensus_queue.cc:1035] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:54.549394 2905 raft_consensus.cc:2932] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:54.554057 27605 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 00205a461b704a19bbd39bc71361e529 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:54.561489 27758 consensus_queue.cc:237] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:12:54.564036 2967 tablet_bootstrap.cc:492] T 4e58bc2dffad423f942ce50144030da4 P f1525cbc82a24d13b65f136695c903af: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:54.565846 30800 raft_consensus.cc:1261] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:12:54.565569 30670 raft_consensus.cc:2932] T 00205a461b704a19bbd39bc71361e529 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:54.569602 2967 tablet_bootstrap.cc:492] T 4e58bc2dffad423f942ce50144030da4 P f1525cbc82a24d13b65f136695c903af: Bootstrap complete.
I20240501 08:12:54.570611 2967 ts_tablet_manager.cc:1367] T 4e58bc2dffad423f942ce50144030da4 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.056s user 0.007s sys 0.001s
I20240501 08:12:54.573917 27620 catalog_manager.cc:5484] T 00205a461b704a19bbd39bc71361e529 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:54.571349 2967 raft_consensus.cc:348] T 4e58bc2dffad423f942ce50144030da4 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:54.577345 2967 raft_consensus.cc:729] T 4e58bc2dffad423f942ce50144030da4 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: LEARNER
I20240501 08:12:54.577584 2967 consensus_queue.cc:260] T 4e58bc2dffad423f942ce50144030da4 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:54.580003 27758 raft_consensus.cc:2932] T 00205a461b704a19bbd39bc71361e529 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:54.583871 2967 ts_tablet_manager.cc:1398] T 4e58bc2dffad423f942ce50144030da4 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240501 08:12:54.586900 30957 tablet_copy_service.cc:342] P a29cd590999b4f32849ff5689cd53c11: Request end of tablet copy session f1525cbc82a24d13b65f136695c903af-4e58bc2dffad423f942ce50144030da4 received from {username='slave'} at 127.21.84.130:49339
I20240501 08:12:54.587090 30957 tablet_copy_service.cc:433] P a29cd590999b4f32849ff5689cd53c11: ending tablet copy session f1525cbc82a24d13b65f136695c903af-4e58bc2dffad423f942ce50144030da4 on tablet 4e58bc2dffad423f942ce50144030da4 with peer f1525cbc82a24d13b65f136695c903af
I20240501 08:12:54.587853 2902 consensus_queue.cc:1035] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:54.596259 30916 tablet_service.cc:1514] Processing DeleteTablet for tablet 00205a461b704a19bbd39bc71361e529 with delete_type TABLET_DATA_TOMBSTONED (TS a29cd590999b4f32849ff5689cd53c11 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54036
I20240501 08:12:54.611093 2674 raft_consensus.cc:2932] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:54.622737 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 73f40a7ef6954b34a48d8c5aeeaebc0a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:54.625788 3007 tablet_replica.cc:330] T 00205a461b704a19bbd39bc71361e529 P a29cd590999b4f32849ff5689cd53c11: stopping tablet replica
I20240501 08:12:54.626120 3007 raft_consensus.cc:2229] T 00205a461b704a19bbd39bc71361e529 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:54.625902 30803 raft_consensus.cc:2932] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:54.626513 3007 raft_consensus.cc:2258] T 00205a461b704a19bbd39bc71361e529 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:54.632611 2992 tablet_replica.cc:330] T a1fde6b765374df9bc4674a5d678c920 P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:54.633106 2992 raft_consensus.cc:2229] T a1fde6b765374df9bc4674a5d678c920 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:54.633404 2992 raft_consensus.cc:2258] T a1fde6b765374df9bc4674a5d678c920 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:54.635289 30537 raft_consensus.cc:1261] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:12:54.635967 27611 catalog_manager.cc:5484] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index 3 to 5, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New cstate: current_term: 2 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:54.637547 2902 consensus_queue.cc:1035] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:54.638999 30537 raft_consensus.cc:2932] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:54.640010 2992 ts_tablet_manager.cc:1875] T a1fde6b765374df9bc4674a5d678c920 P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:54.645591 2976 tablet_copy_client.cc:308] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.130:36503
I20240501 08:12:54.646373 27912 tablet_copy_service.cc:140] P f1525cbc82a24d13b65f136695c903af: Received BeginTabletCopySession request for tablet 728616fd583a4afea00e8fe52880e55b from peer 64998323a1f4482d8704bf20594d13a5 ({username='slave'} at 127.21.84.135:48367)
I20240501 08:12:54.646517 27912 tablet_copy_service.cc:161] P f1525cbc82a24d13b65f136695c903af: Beginning new tablet copy session on tablet 728616fd583a4afea00e8fe52880e55b from peer 64998323a1f4482d8704bf20594d13a5 at {username='slave'} at 127.21.84.135:48367: session id = 64998323a1f4482d8704bf20594d13a5-728616fd583a4afea00e8fe52880e55b
I20240501 08:12:54.649315 27912 tablet_copy_source_session.cc:205] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:54.653065 3007 ts_tablet_manager.cc:1875] T 00205a461b704a19bbd39bc71361e529 P a29cd590999b4f32849ff5689cd53c11: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:54.658157 2976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 728616fd583a4afea00e8fe52880e55b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:54.658275 28152 consensus_queue.cc:237] T c5f4c78e69fb4c67ad65a7b0c1466798 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:12:54.662580 28022 raft_consensus.cc:1261] T c5f4c78e69fb4c67ad65a7b0c1466798 P aad81047009a4ffcbc450b60030583b5 [term 2 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:12:54.662647 30534 raft_consensus.cc:1261] T c5f4c78e69fb4c67ad65a7b0c1466798 P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:12:54.663195 2222 consensus_queue.cc:1035] T c5f4c78e69fb4c67ad65a7b0c1466798 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:54.663419 2607 consensus_queue.cc:1035] T c5f4c78e69fb4c67ad65a7b0c1466798 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:54.669200 28000 tablet_service.cc:1514] Processing DeleteTablet for tablet 73f40a7ef6954b34a48d8c5aeeaebc0a with delete_type TABLET_DATA_TOMBSTONED (TS aad81047009a4ffcbc450b60030583b5 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:54.679531 2977 tablet_copy_client.cc:308] T 462c892043d8475c99389c6580a66f0c P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.134:39933
I20240501 08:12:54.680977 30693 tablet_copy_service.cc:140] P 9a022e1a501f4d44a0468054d6c3de47: Received BeginTabletCopySession request for tablet 462c892043d8475c99389c6580a66f0c from peer 6331f77dae9d452184e8cd0cbf5f1b43 ({username='slave'} at 127.21.84.129:59569)
I20240501 08:12:54.681197 30693 tablet_copy_service.cc:161] P 9a022e1a501f4d44a0468054d6c3de47: Beginning new tablet copy session on tablet 462c892043d8475c99389c6580a66f0c from peer 6331f77dae9d452184e8cd0cbf5f1b43 at {username='slave'} at 127.21.84.129:59569: session id = 6331f77dae9d452184e8cd0cbf5f1b43-462c892043d8475c99389c6580a66f0c
I20240501 08:12:54.684425 30693 tablet_copy_source_session.cc:205] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:54.687723 2977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 462c892043d8475c99389c6580a66f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:54.697439 30534 raft_consensus.cc:2932] T c5f4c78e69fb4c67ad65a7b0c1466798 P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:54.696724 2222 raft_consensus.cc:2932] T c5f4c78e69fb4c67ad65a7b0c1466798 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:54.706382 27611 catalog_manager.cc:5484] T c5f4c78e69fb4c67ad65a7b0c1466798 P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index 3 to 5, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New cstate: current_term: 2 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:54.713568 3007 ts_tablet_manager.cc:1888] T 00205a461b704a19bbd39bc71361e529 P a29cd590999b4f32849ff5689cd53c11: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:54.714951 30784 tablet_service.cc:1514] Processing DeleteTablet for tablet c5f4c78e69fb4c67ad65a7b0c1466798 with delete_type TABLET_DATA_TOMBSTONED (TS 64998323a1f4482d8704bf20594d13a5 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:38952
I20240501 08:12:54.715417 2952 tablet_replica.cc:330] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5: stopping tablet replica
I20240501 08:12:54.715659 2952 raft_consensus.cc:2229] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:54.715844 2952 raft_consensus.cc:2258] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:54.717034 2952 ts_tablet_manager.cc:1875] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:54.725431 2952 ts_tablet_manager.cc:1888] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240501 08:12:54.725680 2952 log.cc:1196] T c5f4c78e69fb4c67ad65a7b0c1466798 P 64998323a1f4482d8704bf20594d13a5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal/wals/c5f4c78e69fb4c67ad65a7b0c1466798
I20240501 08:12:54.726634 27605 catalog_manager.cc:4830] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): tablet c5f4c78e69fb4c67ad65a7b0c1466798 (table rebalancer_extra_table_20 [id=7884fe25785d4c7e9d8b2559f8e81777]) successfully deleted
I20240501 08:12:54.731195 2976 tablet_copy_client.cc:783] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:54.731503 2976 tablet_copy_client.cc:647] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:54.731678 3007 log.cc:1196] T 00205a461b704a19bbd39bc71361e529 P a29cd590999b4f32849ff5689cd53c11: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/wals/00205a461b704a19bbd39bc71361e529
I20240501 08:12:54.735805 2976 tablet_copy_client.cc:523] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:54.737361 27606 catalog_manager.cc:4830] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): tablet 00205a461b704a19bbd39bc71361e529 (table rebalancer_extra_table_60 [id=9a256e3f732a4230901aea19c32440ba]) successfully deleted
I20240501 08:12:54.745793 2977 tablet_copy_client.cc:783] T 462c892043d8475c99389c6580a66f0c P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:54.752681 3015 tablet_replica.cc:330] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:54.753015 3015 raft_consensus.cc:2229] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:54.753234 3015 pending_rounds.cc:62] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5: Trying to abort 1 pending ops.
I20240501 08:12:54.753388 3015 pending_rounds.cc:68] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5: Aborting op as it isn't in flight: id { term: 2 index: 4 } timestamp: 7022801608165036032 op_type: NO_OP noop_request { }
I20240501 08:12:54.753698 3015 raft_consensus.cc:2866] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240501 08:12:54.753984 3015 raft_consensus.cc:2258] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:54.754521 2977 tablet_copy_client.cc:647] T 462c892043d8475c99389c6580a66f0c P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:54.773494 2977 tablet_copy_client.cc:523] T 462c892043d8475c99389c6580a66f0c P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:54.780591 2976 tablet_bootstrap.cc:492] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:54.791002 3015 ts_tablet_manager.cc:1875] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:54.807988 3015 ts_tablet_manager.cc:1888] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240501 08:12:54.808326 3015 log.cc:1196] T 73f40a7ef6954b34a48d8c5aeeaebc0a P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/73f40a7ef6954b34a48d8c5aeeaebc0a
I20240501 08:12:54.809504 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet 73f40a7ef6954b34a48d8c5aeeaebc0a (table rebalancer_extra_table_30 [id=9da53f67716d4d2892b2ff2b43922ec5]) successfully deleted
I20240501 08:12:54.817888 27604 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c5f4c78e69fb4c67ad65a7b0c1466798 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240501 08:12:54.894052 28087 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.131:36305, user_credentials={real_user=slave}} blocked reactor thread for 70896.8us
I20240501 08:12:54.910290 2977 tablet_bootstrap.cc:492] T 462c892043d8475c99389c6580a66f0c P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:54.930965 27891 raft_consensus.cc:1203] T 4e58bc2dffad423f942ce50144030da4 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:54.965447 2984 tablet_copy_client.cc:308] T b6ad187384514267b57cc994c9612709 P aad81047009a4ffcbc450b60030583b5: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.135:33503
I20240501 08:12:54.966252 30825 tablet_copy_service.cc:140] P 64998323a1f4482d8704bf20594d13a5: Received BeginTabletCopySession request for tablet b6ad187384514267b57cc994c9612709 from peer aad81047009a4ffcbc450b60030583b5 ({username='slave'} at 127.21.84.131:43681)
I20240501 08:12:54.966455 30825 tablet_copy_service.cc:161] P 64998323a1f4482d8704bf20594d13a5: Beginning new tablet copy session on tablet b6ad187384514267b57cc994c9612709 from peer aad81047009a4ffcbc450b60030583b5 at {username='slave'} at 127.21.84.131:43681: session id = aad81047009a4ffcbc450b60030583b5-b6ad187384514267b57cc994c9612709
I20240501 08:12:54.970333 30825 tablet_copy_source_session.cc:205] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:54.972453 28020 raft_consensus.cc:2932] T c5f4c78e69fb4c67ad65a7b0c1466798 P aad81047009a4ffcbc450b60030583b5 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:12:54.976215 30784 tablet_service.cc:1514] Processing DeleteTablet for tablet c5f4c78e69fb4c67ad65a7b0c1466798 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 3 (current committed config index is 5)) from {username='slave'} at 127.0.0.1:38952
I20240501 08:12:54.977072 27605 catalog_manager.cc:4830] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): tablet c5f4c78e69fb4c67ad65a7b0c1466798 (table rebalancer_extra_table_20 [id=7884fe25785d4c7e9d8b2559f8e81777]) successfully deleted
I20240501 08:12:54.978099 2984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6ad187384514267b57cc994c9612709. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:54.998919 2984 tablet_copy_client.cc:783] T b6ad187384514267b57cc994c9612709 P aad81047009a4ffcbc450b60030583b5: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:54.999239 2984 tablet_copy_client.cc:647] T b6ad187384514267b57cc994c9612709 P aad81047009a4ffcbc450b60030583b5: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:55.001127 2984 tablet_copy_client.cc:523] T b6ad187384514267b57cc994c9612709 P aad81047009a4ffcbc450b60030583b5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:55.008842 2984 tablet_bootstrap.cc:492] T b6ad187384514267b57cc994c9612709 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:55.016765 2976 tablet_bootstrap.cc:492] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:55.017270 2976 tablet_bootstrap.cc:492] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5: Bootstrap complete.
I20240501 08:12:55.017570 2976 ts_tablet_manager.cc:1367] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.237s user 0.052s sys 0.007s
I20240501 08:12:55.018522 2976 raft_consensus.cc:348] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
I20240501 08:12:55.019275 2976 raft_consensus.cc:729] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: LEARNER
I20240501 08:12:55.019472 2976 consensus_queue.cc:260] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
I20240501 08:12:55.024317 2976 ts_tablet_manager.cc:1398] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240501 08:12:55.025128 27912 tablet_copy_service.cc:342] P f1525cbc82a24d13b65f136695c903af: Request end of tablet copy session 64998323a1f4482d8704bf20594d13a5-728616fd583a4afea00e8fe52880e55b received from {username='slave'} at 127.21.84.135:48367
I20240501 08:12:55.025251 27912 tablet_copy_service.cc:433] P f1525cbc82a24d13b65f136695c903af: ending tablet copy session 64998323a1f4482d8704bf20594d13a5-728616fd583a4afea00e8fe52880e55b on tablet 728616fd583a4afea00e8fe52880e55b with peer 64998323a1f4482d8704bf20594d13a5
I20240501 08:12:55.030853 2984 tablet_bootstrap.cc:492] T b6ad187384514267b57cc994c9612709 P aad81047009a4ffcbc450b60030583b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:55.031461 2984 tablet_bootstrap.cc:492] T b6ad187384514267b57cc994c9612709 P aad81047009a4ffcbc450b60030583b5: Bootstrap complete.
I20240501 08:12:55.031813 2984 ts_tablet_manager.cc:1367] T b6ad187384514267b57cc994c9612709 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.005s
I20240501 08:12:55.032877 2984 raft_consensus.cc:348] T b6ad187384514267b57cc994c9612709 P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:12:55.033286 2984 raft_consensus.cc:729] T b6ad187384514267b57cc994c9612709 P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: LEARNER
I20240501 08:12:55.033551 2984 consensus_queue.cc:260] T b6ad187384514267b57cc994c9612709 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:12:55.034363 2984 ts_tablet_manager.cc:1398] T b6ad187384514267b57cc994c9612709 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:55.036219 30825 tablet_copy_service.cc:342] P 64998323a1f4482d8704bf20594d13a5: Request end of tablet copy session aad81047009a4ffcbc450b60030583b5-b6ad187384514267b57cc994c9612709 received from {username='slave'} at 127.21.84.131:43681
I20240501 08:12:55.036347 30825 tablet_copy_service.cc:433] P 64998323a1f4482d8704bf20594d13a5: ending tablet copy session aad81047009a4ffcbc450b60030583b5-b6ad187384514267b57cc994c9612709 on tablet b6ad187384514267b57cc994c9612709 with peer aad81047009a4ffcbc450b60030583b5
I20240501 08:12:55.063158 2992 ts_tablet_manager.cc:1888] T a1fde6b765374df9bc4674a5d678c920 P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:55.064456 2992 log.cc:1196] T a1fde6b765374df9bc4674a5d678c920 P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/a1fde6b765374df9bc4674a5d678c920
I20240501 08:12:55.070411 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet a1fde6b765374df9bc4674a5d678c920 (table rebalancer_extra_table_48 [id=e75e40faa06f4299a600ba7063fa7b4b]) successfully deleted
I20240501 08:12:55.073302 2977 tablet_bootstrap.cc:492] T 462c892043d8475c99389c6580a66f0c P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:55.073688 2977 tablet_bootstrap.cc:492] T 462c892043d8475c99389c6580a66f0c P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap complete.
I20240501 08:12:55.073969 2977 ts_tablet_manager.cc:1367] T 462c892043d8475c99389c6580a66f0c P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.295s user 0.003s sys 0.004s
I20240501 08:12:55.074644 2977 raft_consensus.cc:348] T 462c892043d8475c99389c6580a66f0c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:12:55.075009 2977 raft_consensus.cc:729] T 462c892043d8475c99389c6580a66f0c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: LEARNER
I20240501 08:12:55.075269 2977 consensus_queue.cc:260] T 462c892043d8475c99389c6580a66f0c P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:12:55.083942 2977 ts_tablet_manager.cc:1398] T 462c892043d8475c99389c6580a66f0c P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.010s user 0.000s sys 0.001s
I20240501 08:12:55.093468 30693 tablet_copy_service.cc:342] P 9a022e1a501f4d44a0468054d6c3de47: Request end of tablet copy session 6331f77dae9d452184e8cd0cbf5f1b43-462c892043d8475c99389c6580a66f0c received from {username='slave'} at 127.21.84.129:59569
I20240501 08:12:55.093605 30693 tablet_copy_service.cc:433] P 9a022e1a501f4d44a0468054d6c3de47: ending tablet copy session 6331f77dae9d452184e8cd0cbf5f1b43-462c892043d8475c99389c6580a66f0c on tablet 462c892043d8475c99389c6580a66f0c with peer 6331f77dae9d452184e8cd0cbf5f1b43
I20240501 08:12:55.139190 27758 raft_consensus.cc:1203] T 462c892043d8475c99389c6580a66f0c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:55.218621 30537 consensus_queue.cc:237] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:55.220842 30672 raft_consensus.cc:1261] T 80bb0e53796f4b319fa9f877451076f3 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:12:55.221045 30473 consensus_peers.cc:485] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80bb0e53796f4b319fa9f877451076f3. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:55.221352 28149 raft_consensus.cc:1261] T 80bb0e53796f4b319fa9f877451076f3 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:55.221393 3078 consensus_queue.cc:1035] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:55.221822 2594 consensus_queue.cc:1035] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:55.224715 2594 raft_consensus.cc:2932] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:55.225204 30672 raft_consensus.cc:2932] T 80bb0e53796f4b319fa9f877451076f3 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:55.225593 28149 raft_consensus.cc:2932] T 80bb0e53796f4b319fa9f877451076f3 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:55.234269 27619 catalog_manager.cc:5484] T 80bb0e53796f4b319fa9f877451076f3 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index -1 to 2, NON_VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) added. New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } } }
I20240501 08:12:55.270759 30537 consensus_queue.cc:237] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
W20240501 08:12:55.273523 30472 consensus_peers.cc:485] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e12867cd76504438abe91fbe15498230. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:55.274647 27891 raft_consensus.cc:1261] T e12867cd76504438abe91fbe15498230 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:55.275110 30672 raft_consensus.cc:1261] T e12867cd76504438abe91fbe15498230 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:55.275195 3078 consensus_queue.cc:1035] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:55.275586 2594 consensus_queue.cc:1035] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:55.281139 3078 raft_consensus.cc:2932] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:55.285761 27891 raft_consensus.cc:2932] T e12867cd76504438abe91fbe15498230 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:55.303670 27615 catalog_manager.cc:5484] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:55.312259 30672 raft_consensus.cc:2932] T e12867cd76504438abe91fbe15498230 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:55.350231 28149 consensus_queue.cc:237] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
W20240501 08:12:55.352767 28086 consensus_peers.cc:485] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 -> Peer 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): Couldn't send request to peer 64998323a1f4482d8704bf20594d13a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8eb9763a1573455287b7137fe7fecf62. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:55.352865 27891 raft_consensus.cc:1261] T 8eb9763a1573455287b7137fe7fecf62 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:55.352882 28020 raft_consensus.cc:1261] T 8eb9763a1573455287b7137fe7fecf62 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:55.353438 2222 consensus_queue.cc:1035] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:55.355181 2222 consensus_queue.cc:1035] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:55.356734 2222 raft_consensus.cc:2932] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:12:55.358850 28020 raft_consensus.cc:2932] T 8eb9763a1573455287b7137fe7fecf62 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:12:55.360595 27891 raft_consensus.cc:2932] T 8eb9763a1573455287b7137fe7fecf62 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:12:55.365721 27615 catalog_manager.cc:5484] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: config changed from index -1 to 4, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:55.409847 27889 consensus_queue.cc:237] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:12:55.416697 28020 raft_consensus.cc:1261] T 24825c85238b4d618c65d00107135420 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:12:55.417019 27827 consensus_peers.cc:485] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af -> Peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): Couldn't send request to peer a29cd590999b4f32849ff5689cd53c11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 24825c85238b4d618c65d00107135420. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:55.417331 2786 consensus_queue.cc:1035] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:55.418874 2602 raft_consensus.cc:1050] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11: attempting to promote NON_VOTER f1525cbc82a24d13b65f136695c903af to VOTER
I20240501 08:12:55.419689 2602 consensus_queue.cc:237] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:55.420894 27758 raft_consensus.cc:1261] T 24825c85238b4d618c65d00107135420 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:55.421504 2786 consensus_queue.cc:1035] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:55.426782 2786 raft_consensus.cc:2932] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:55.433281 28020 raft_consensus.cc:2932] T 24825c85238b4d618c65d00107135420 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:55.434136 27758 raft_consensus.cc:2932] T 24825c85238b4d618c65d00107135420 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:55.446403 30672 raft_consensus.cc:1261] T 4e58bc2dffad423f942ce50144030da4 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:55.447496 27618 catalog_manager.cc:5484] T 24825c85238b4d618c65d00107135420 P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index -1 to 4, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:55.448159 30537 raft_consensus.cc:1261] T 4e58bc2dffad423f942ce50144030da4 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:55.449405 2755 consensus_queue.cc:1035] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:55.453382 27889 raft_consensus.cc:1261] T 4e58bc2dffad423f942ce50144030da4 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:55.456738 2755 consensus_queue.cc:1035] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:55.457597 2755 consensus_queue.cc:1035] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:55.472213 2755 raft_consensus.cc:2932] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:55.482545 30536 raft_consensus.cc:2932] T 4e58bc2dffad423f942ce50144030da4 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:55.485343 27620 catalog_manager.cc:5484] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:55.491878 28020 raft_consensus.cc:1203] T b6ad187384514267b57cc994c9612709 P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:55.491825 27889 raft_consensus.cc:2932] T 4e58bc2dffad423f942ce50144030da4 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:55.508881 30670 raft_consensus.cc:2932] T 4e58bc2dffad423f942ce50144030da4 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:55.519691 30936 consensus_queue.cc:237] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:55.524613 27889 raft_consensus.cc:1261] T 4e58bc2dffad423f942ce50144030da4 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:55.527698 3026 consensus_queue.cc:1035] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:55.529227 30537 raft_consensus.cc:1261] T 4e58bc2dffad423f942ce50144030da4 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:55.530061 3026 consensus_queue.cc:1035] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:55.533094 2755 raft_consensus.cc:2932] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:55.534420 30803 raft_consensus.cc:1203] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:55.539199 30537 raft_consensus.cc:2932] T 4e58bc2dffad423f942ce50144030da4 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:55.543269 27889 raft_consensus.cc:2932] T 4e58bc2dffad423f942ce50144030da4 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:55.546298 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4e58bc2dffad423f942ce50144030da4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:55.550379 27620 catalog_manager.cc:5484] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:55.559031 30650 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e58bc2dffad423f942ce50144030da4 with delete_type TABLET_DATA_TOMBSTONED (TS 9a022e1a501f4d44a0468054d6c3de47 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57156
I20240501 08:12:55.562938 3123 tablet_replica.cc:330] T 4e58bc2dffad423f942ce50144030da4 P 9a022e1a501f4d44a0468054d6c3de47: stopping tablet replica
I20240501 08:12:55.564349 3123 raft_consensus.cc:2229] T 4e58bc2dffad423f942ce50144030da4 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:55.564810 3123 raft_consensus.cc:2258] T 4e58bc2dffad423f942ce50144030da4 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:55.566133 3123 ts_tablet_manager.cc:1875] T 4e58bc2dffad423f942ce50144030da4 P 9a022e1a501f4d44a0468054d6c3de47: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:55.568306 27758 consensus_queue.cc:237] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:12:55.571166 28020 raft_consensus.cc:1261] T f3b6691cbe8740d2978c20209c78f3ff P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:55.571202 28149 raft_consensus.cc:1261] T f3b6691cbe8740d2978c20209c78f3ff P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:55.571900 2902 consensus_queue.cc:1035] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:55.572407 2674 consensus_queue.cc:1035] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:12:55.576151 27696 consensus_peers.cc:485] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 -> Peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): Couldn't send request to peer a29cd590999b4f32849ff5689cd53c11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3b6691cbe8740d2978c20209c78f3ff. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:55.580336 2933 raft_consensus.cc:2932] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:55.580894 28149 raft_consensus.cc:2932] T f3b6691cbe8740d2978c20209c78f3ff P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:55.588078 27620 catalog_manager.cc:5484] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index -1 to 4, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:55.593864 3123 ts_tablet_manager.cc:1888] T 4e58bc2dffad423f942ce50144030da4 P 9a022e1a501f4d44a0468054d6c3de47: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:55.597579 3123 log.cc:1196] T 4e58bc2dffad423f942ce50144030da4 P 9a022e1a501f4d44a0468054d6c3de47: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/wals/4e58bc2dffad423f942ce50144030da4
I20240501 08:12:55.598839 27604 catalog_manager.cc:4830] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): tablet 4e58bc2dffad423f942ce50144030da4 (table rebalancer_extra_table_36 [id=a0611396e39b4372a1b4597945f4a169]) successfully deleted
I20240501 08:12:55.605396 28016 raft_consensus.cc:2932] T f3b6691cbe8740d2978c20209c78f3ff P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:55.619647 27889 consensus_queue.cc:237] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
W20240501 08:12:55.626053 27827 consensus_peers.cc:485] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af -> Peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): Couldn't send request to peer a29cd590999b4f32849ff5689cd53c11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a2981c0863bc41729605f8b076b6a6cb. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:55.627103 28149 raft_consensus.cc:1261] T a2981c0863bc41729605f8b076b6a6cb P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:55.627879 2786 consensus_queue.cc:1035] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:55.628845 28016 raft_consensus.cc:1261] T a2981c0863bc41729605f8b076b6a6cb P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:55.630437 1679 raft_consensus.cc:1050] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47: attempting to promote NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 to VOTER
I20240501 08:12:55.631348 3113 consensus_queue.cc:1035] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:55.631332 1679 consensus_queue.cc:237] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:12:55.632066 2620 raft_consensus.cc:2932] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:55.633728 28151 raft_consensus.cc:2932] T a2981c0863bc41729605f8b076b6a6cb P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:55.646956 28020 raft_consensus.cc:2932] T a2981c0863bc41729605f8b076b6a6cb P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:12:55.647262 27615 catalog_manager.cc:5484] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af reported cstate change: config changed from index -1 to 4, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:55.662453 27758 raft_consensus.cc:1261] T 462c892043d8475c99389c6580a66f0c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:55.663005 30936 raft_consensus.cc:1261] T 462c892043d8475c99389c6580a66f0c P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:55.663556 1679 consensus_queue.cc:1035] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:55.664142 30803 raft_consensus.cc:1261] T 462c892043d8475c99389c6580a66f0c P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:55.664690 3049 consensus_queue.cc:1035] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:55.666046 3049 consensus_queue.cc:1035] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:55.676064 1679 raft_consensus.cc:2932] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:55.677284 27760 raft_consensus.cc:2932] T 462c892043d8475c99389c6580a66f0c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:55.682660 30936 raft_consensus.cc:2932] T 462c892043d8475c99389c6580a66f0c P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:55.685014 27620 catalog_manager.cc:5484] T 462c892043d8475c99389c6580a66f0c P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:55.696018 30803 raft_consensus.cc:2932] T 462c892043d8475c99389c6580a66f0c P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:55.705129 30670 consensus_queue.cc:237] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:12:55.707258 27760 raft_consensus.cc:1261] T 462c892043d8475c99389c6580a66f0c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:55.709017 1679 consensus_queue.cc:1035] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:55.713667 2436 raft_consensus.cc:2932] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:55.717077 27604 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 462c892043d8475c99389c6580a66f0c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:55.718871 27758 raft_consensus.cc:2932] T 462c892043d8475c99389c6580a66f0c P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:55.722597 30801 raft_consensus.cc:1261] T 462c892043d8475c99389c6580a66f0c P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:55.723836 3049 consensus_queue.cc:1035] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:55.727054 30803 raft_consensus.cc:2932] T 462c892043d8475c99389c6580a66f0c P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:55.732383 27618 catalog_manager.cc:5484] T 462c892043d8475c99389c6580a66f0c P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New cstate: current_term: 1 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:55.745373 30916 tablet_service.cc:1514] Processing DeleteTablet for tablet 462c892043d8475c99389c6580a66f0c with delete_type TABLET_DATA_TOMBSTONED (TS a29cd590999b4f32849ff5689cd53c11 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54036
I20240501 08:12:55.750882 3151 tablet_replica.cc:330] T 462c892043d8475c99389c6580a66f0c P a29cd590999b4f32849ff5689cd53c11: stopping tablet replica
I20240501 08:12:55.751109 3151 raft_consensus.cc:2229] T 462c892043d8475c99389c6580a66f0c P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:55.751341 3151 raft_consensus.cc:2258] T 462c892043d8475c99389c6580a66f0c P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:55.754796 3151 ts_tablet_manager.cc:1875] T 462c892043d8475c99389c6580a66f0c P a29cd590999b4f32849ff5689cd53c11: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:55.760948 3151 ts_tablet_manager.cc:1888] T 462c892043d8475c99389c6580a66f0c P a29cd590999b4f32849ff5689cd53c11: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:55.761242 3151 log.cc:1196] T 462c892043d8475c99389c6580a66f0c P a29cd590999b4f32849ff5689cd53c11: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/wals/462c892043d8475c99389c6580a66f0c
I20240501 08:12:55.762626 27606 catalog_manager.cc:4830] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): tablet 462c892043d8475c99389c6580a66f0c (table rebalancer_extra_table_36 [id=a0611396e39b4372a1b4597945f4a169]) successfully deleted
I20240501 08:12:55.773072 3157 ts_tablet_manager.cc:902] T e12867cd76504438abe91fbe15498230 P 6331f77dae9d452184e8cd0cbf5f1b43: Initiating tablet copy from peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:55.802332 3157 tablet_copy_client.cc:308] T e12867cd76504438abe91fbe15498230 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.133:38293
I20240501 08:12:55.803485 30559 tablet_copy_service.cc:140] P 31811708c6774037b82df44024b229b6: Received BeginTabletCopySession request for tablet e12867cd76504438abe91fbe15498230 from peer 6331f77dae9d452184e8cd0cbf5f1b43 ({username='slave'} at 127.21.84.129:35413)
I20240501 08:12:55.803697 30559 tablet_copy_service.cc:161] P 31811708c6774037b82df44024b229b6: Beginning new tablet copy session on tablet e12867cd76504438abe91fbe15498230 from peer 6331f77dae9d452184e8cd0cbf5f1b43 at {username='slave'} at 127.21.84.129:35413: session id = 6331f77dae9d452184e8cd0cbf5f1b43-e12867cd76504438abe91fbe15498230
I20240501 08:12:55.806133 30559 tablet_copy_source_session.cc:205] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:55.807613 3157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e12867cd76504438abe91fbe15498230. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:55.810940 3168 ts_tablet_manager.cc:902] T 24825c85238b4d618c65d00107135420 P a29cd590999b4f32849ff5689cd53c11: Initiating tablet copy from peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:12:55.813854 3168 tablet_copy_client.cc:308] T 24825c85238b4d618c65d00107135420 P a29cd590999b4f32849ff5689cd53c11: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.130:36503
I20240501 08:12:55.814553 3157 tablet_copy_client.cc:783] T e12867cd76504438abe91fbe15498230 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:55.814657 27912 tablet_copy_service.cc:140] P f1525cbc82a24d13b65f136695c903af: Received BeginTabletCopySession request for tablet 24825c85238b4d618c65d00107135420 from peer a29cd590999b4f32849ff5689cd53c11 ({username='slave'} at 127.21.84.136:54747)
I20240501 08:12:55.814785 3157 tablet_copy_client.cc:647] T e12867cd76504438abe91fbe15498230 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:55.814813 27912 tablet_copy_service.cc:161] P f1525cbc82a24d13b65f136695c903af: Beginning new tablet copy session on tablet 24825c85238b4d618c65d00107135420 from peer a29cd590999b4f32849ff5689cd53c11 at {username='slave'} at 127.21.84.136:54747: session id = a29cd590999b4f32849ff5689cd53c11-24825c85238b4d618c65d00107135420
I20240501 08:12:55.816471 3157 tablet_copy_client.cc:523] T e12867cd76504438abe91fbe15498230 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:55.817173 27912 tablet_copy_source_session.cc:205] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:55.819129 3157 tablet_bootstrap.cc:492] T e12867cd76504438abe91fbe15498230 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:55.820233 3168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24825c85238b4d618c65d00107135420. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:55.824640 3157 tablet_bootstrap.cc:492] T e12867cd76504438abe91fbe15498230 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:55.825304 3157 tablet_bootstrap.cc:492] T e12867cd76504438abe91fbe15498230 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap complete.
I20240501 08:12:55.825529 3157 ts_tablet_manager.cc:1367] T e12867cd76504438abe91fbe15498230 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20240501 08:12:55.826196 3157 raft_consensus.cc:348] T e12867cd76504438abe91fbe15498230 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:12:55.826434 3157 raft_consensus.cc:729] T e12867cd76504438abe91fbe15498230 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: LEARNER
I20240501 08:12:55.826613 3157 consensus_queue.cc:260] T e12867cd76504438abe91fbe15498230 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:12:55.827322 3157 ts_tablet_manager.cc:1398] T e12867cd76504438abe91fbe15498230 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:55.828195 30559 tablet_copy_service.cc:342] P 31811708c6774037b82df44024b229b6: Request end of tablet copy session 6331f77dae9d452184e8cd0cbf5f1b43-e12867cd76504438abe91fbe15498230 received from {username='slave'} at 127.21.84.129:35413
I20240501 08:12:55.828352 30559 tablet_copy_service.cc:433] P 31811708c6774037b82df44024b229b6: ending tablet copy session 6331f77dae9d452184e8cd0cbf5f1b43-e12867cd76504438abe91fbe15498230 on tablet e12867cd76504438abe91fbe15498230 with peer 6331f77dae9d452184e8cd0cbf5f1b43
I20240501 08:12:55.837101 3168 tablet_copy_client.cc:783] T 24825c85238b4d618c65d00107135420 P a29cd590999b4f32849ff5689cd53c11: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:55.840489 3168 tablet_copy_client.cc:647] T 24825c85238b4d618c65d00107135420 P a29cd590999b4f32849ff5689cd53c11: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:55.845623 3168 tablet_copy_client.cc:523] T 24825c85238b4d618c65d00107135420 P a29cd590999b4f32849ff5689cd53c11: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:55.846401 3178 ts_tablet_manager.cc:902] T 80bb0e53796f4b319fa9f877451076f3 P f1525cbc82a24d13b65f136695c903af: Initiating tablet copy from peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:55.860134 3178 tablet_copy_client.cc:308] T 80bb0e53796f4b319fa9f877451076f3 P f1525cbc82a24d13b65f136695c903af: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.133:38293
I20240501 08:12:55.861056 30559 tablet_copy_service.cc:140] P 31811708c6774037b82df44024b229b6: Received BeginTabletCopySession request for tablet 80bb0e53796f4b319fa9f877451076f3 from peer f1525cbc82a24d13b65f136695c903af ({username='slave'} at 127.21.84.130:58953)
I20240501 08:12:55.861238 30559 tablet_copy_service.cc:161] P 31811708c6774037b82df44024b229b6: Beginning new tablet copy session on tablet 80bb0e53796f4b319fa9f877451076f3 from peer f1525cbc82a24d13b65f136695c903af at {username='slave'} at 127.21.84.130:58953: session id = f1525cbc82a24d13b65f136695c903af-80bb0e53796f4b319fa9f877451076f3
I20240501 08:12:55.863708 30559 tablet_copy_source_session.cc:205] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:55.866235 3178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80bb0e53796f4b319fa9f877451076f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:55.867427 3168 tablet_bootstrap.cc:492] T 24825c85238b4d618c65d00107135420 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:55.873654 3178 tablet_copy_client.cc:783] T 80bb0e53796f4b319fa9f877451076f3 P f1525cbc82a24d13b65f136695c903af: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:55.873953 3178 tablet_copy_client.cc:647] T 80bb0e53796f4b319fa9f877451076f3 P f1525cbc82a24d13b65f136695c903af: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:55.877203 3178 tablet_copy_client.cc:523] T 80bb0e53796f4b319fa9f877451076f3 P f1525cbc82a24d13b65f136695c903af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:55.881794 3178 tablet_bootstrap.cc:492] T 80bb0e53796f4b319fa9f877451076f3 P f1525cbc82a24d13b65f136695c903af: Bootstrap starting.
I20240501 08:12:55.890502 2226 raft_consensus.cc:1050] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5: attempting to promote NON_VOTER aad81047009a4ffcbc450b60030583b5 to VOTER
I20240501 08:12:55.891464 2226 consensus_queue.cc:237] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } }
I20240501 08:12:55.911398 28016 raft_consensus.cc:1261] T b6ad187384514267b57cc994c9612709 P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:55.912019 28151 raft_consensus.cc:1261] T b6ad187384514267b57cc994c9612709 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:55.912925 3117 consensus_queue.cc:1035] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:55.916098 3117 consensus_queue.cc:1035] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:55.917788 3178 tablet_bootstrap.cc:492] T 80bb0e53796f4b319fa9f877451076f3 P f1525cbc82a24d13b65f136695c903af: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:55.918255 3178 tablet_bootstrap.cc:492] T 80bb0e53796f4b319fa9f877451076f3 P f1525cbc82a24d13b65f136695c903af: Bootstrap complete.
I20240501 08:12:55.918512 3178 ts_tablet_manager.cc:1367] T 80bb0e53796f4b319fa9f877451076f3 P f1525cbc82a24d13b65f136695c903af: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.011s
I20240501 08:12:55.919690 3178 raft_consensus.cc:348] T 80bb0e53796f4b319fa9f877451076f3 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:55.919912 3178 raft_consensus.cc:729] T 80bb0e53796f4b319fa9f877451076f3 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Initialized, Role: LEARNER
I20240501 08:12:55.920118 3178 consensus_queue.cc:260] T 80bb0e53796f4b319fa9f877451076f3 P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: NON_VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: true } }
I20240501 08:12:55.920707 3178 ts_tablet_manager.cc:1398] T 80bb0e53796f4b319fa9f877451076f3 P f1525cbc82a24d13b65f136695c903af: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:12:55.921391 30559 tablet_copy_service.cc:342] P 31811708c6774037b82df44024b229b6: Request end of tablet copy session f1525cbc82a24d13b65f136695c903af-80bb0e53796f4b319fa9f877451076f3 received from {username='slave'} at 127.21.84.130:58953
I20240501 08:12:55.921514 30559 tablet_copy_service.cc:433] P 31811708c6774037b82df44024b229b6: ending tablet copy session f1525cbc82a24d13b65f136695c903af-80bb0e53796f4b319fa9f877451076f3 on tablet 80bb0e53796f4b319fa9f877451076f3 with peer f1525cbc82a24d13b65f136695c903af
I20240501 08:12:55.927551 3195 ts_tablet_manager.cc:902] T 8eb9763a1573455287b7137fe7fecf62 P 64998323a1f4482d8704bf20594d13a5: Initiating tablet copy from peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:12:55.942885 30936 raft_consensus.cc:1261] T b6ad187384514267b57cc994c9612709 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:55.943646 3117 consensus_queue.cc:1035] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:55.955047 3117 raft_consensus.cc:2932] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:55.960552 28151 raft_consensus.cc:2932] T b6ad187384514267b57cc994c9612709 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:55.966038 27620 catalog_manager.cc:5484] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:55.968565 3168 tablet_bootstrap.cc:492] T 24825c85238b4d618c65d00107135420 P a29cd590999b4f32849ff5689cd53c11: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:55.969007 3168 tablet_bootstrap.cc:492] T 24825c85238b4d618c65d00107135420 P a29cd590999b4f32849ff5689cd53c11: Bootstrap complete.
I20240501 08:12:55.969254 3168 ts_tablet_manager.cc:1367] T 24825c85238b4d618c65d00107135420 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.102s user 0.059s sys 0.001s
I20240501 08:12:55.970206 3168 raft_consensus.cc:348] T 24825c85238b4d618c65d00107135420 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:12:55.970484 3168 raft_consensus.cc:729] T 24825c85238b4d618c65d00107135420 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: LEARNER
I20240501 08:12:55.970737 3168 consensus_queue.cc:260] T 24825c85238b4d618c65d00107135420 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:12:55.971385 3168 ts_tablet_manager.cc:1398] T 24825c85238b4d618c65d00107135420 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 08:12:55.972836 27912 tablet_copy_service.cc:342] P f1525cbc82a24d13b65f136695c903af: Request end of tablet copy session a29cd590999b4f32849ff5689cd53c11-24825c85238b4d618c65d00107135420 received from {username='slave'} at 127.21.84.136:54747
I20240501 08:12:55.972961 27912 tablet_copy_service.cc:433] P f1525cbc82a24d13b65f136695c903af: ending tablet copy session a29cd590999b4f32849ff5689cd53c11-24825c85238b4d618c65d00107135420 on tablet 24825c85238b4d618c65d00107135420 with peer a29cd590999b4f32849ff5689cd53c11
I20240501 08:12:55.978366 3168 ts_tablet_manager.cc:902] T f3b6691cbe8740d2978c20209c78f3ff P a29cd590999b4f32849ff5689cd53c11: Initiating tablet copy from peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:12:55.989113 28020 raft_consensus.cc:2932] T b6ad187384514267b57cc994c9612709 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:55.989696 30934 raft_consensus.cc:2932] T b6ad187384514267b57cc994c9612709 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:56.003126 30801 consensus_queue.cc:237] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } }
I20240501 08:12:56.005656 28016 raft_consensus.cc:1261] T b6ad187384514267b57cc994c9612709 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:56.006487 2226 consensus_queue.cc:1035] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:56.006657 28151 raft_consensus.cc:1261] T b6ad187384514267b57cc994c9612709 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:56.007364 3109 consensus_queue.cc:1035] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:56.009699 3109 raft_consensus.cc:2932] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:56.010846 28151 raft_consensus.cc:2932] T b6ad187384514267b57cc994c9612709 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:56.014294 27605 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b6ad187384514267b57cc994c9612709 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:56.018051 27615 catalog_manager.cc:5484] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:56.028832 28020 raft_consensus.cc:2932] T b6ad187384514267b57cc994c9612709 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:56.034721 30916 tablet_service.cc:1514] Processing DeleteTablet for tablet b6ad187384514267b57cc994c9612709 with delete_type TABLET_DATA_TOMBSTONED (TS a29cd590999b4f32849ff5689cd53c11 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54036
I20240501 08:12:56.035127 3151 tablet_replica.cc:330] T b6ad187384514267b57cc994c9612709 P a29cd590999b4f32849ff5689cd53c11: stopping tablet replica
I20240501 08:12:56.035341 3151 raft_consensus.cc:2229] T b6ad187384514267b57cc994c9612709 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:56.035600 3151 raft_consensus.cc:2258] T b6ad187384514267b57cc994c9612709 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:56.038391 3151 ts_tablet_manager.cc:1875] T b6ad187384514267b57cc994c9612709 P a29cd590999b4f32849ff5689cd53c11: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:56.039669 3195 tablet_copy_client.cc:308] T 8eb9763a1573455287b7137fe7fecf62 P 64998323a1f4482d8704bf20594d13a5: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.132:41323
I20240501 08:12:56.050122 3151 ts_tablet_manager.cc:1888] T b6ad187384514267b57cc994c9612709 P a29cd590999b4f32849ff5689cd53c11: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:56.050395 3151 log.cc:1196] T b6ad187384514267b57cc994c9612709 P a29cd590999b4f32849ff5689cd53c11: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/wals/b6ad187384514267b57cc994c9612709
I20240501 08:12:56.050757 3203 ts_tablet_manager.cc:902] T a2981c0863bc41729605f8b076b6a6cb P a29cd590999b4f32849ff5689cd53c11: Initiating tablet copy from peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:12:56.051519 27606 catalog_manager.cc:4830] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): tablet b6ad187384514267b57cc994c9612709 (table rebalancer_extra_table_12 [id=7c8060b6145d492cadb48df007ce38c9]) successfully deleted
I20240501 08:12:56.073766 28171 tablet_copy_service.cc:140] P 91d11eae5ef6432ca86eb8ed446b6ea6: Received BeginTabletCopySession request for tablet 8eb9763a1573455287b7137fe7fecf62 from peer 64998323a1f4482d8704bf20594d13a5 ({username='slave'} at 127.21.84.135:48917)
I20240501 08:12:56.073971 28171 tablet_copy_service.cc:161] P 91d11eae5ef6432ca86eb8ed446b6ea6: Beginning new tablet copy session on tablet 8eb9763a1573455287b7137fe7fecf62 from peer 64998323a1f4482d8704bf20594d13a5 at {username='slave'} at 127.21.84.135:48917: session id = 64998323a1f4482d8704bf20594d13a5-8eb9763a1573455287b7137fe7fecf62
I20240501 08:12:56.076817 28171 tablet_copy_source_session.cc:205] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:56.086130 3168 tablet_copy_client.cc:308] T f3b6691cbe8740d2978c20209c78f3ff P a29cd590999b4f32849ff5689cd53c11: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.129:43471
I20240501 08:12:56.086918 27781 tablet_copy_service.cc:140] P 6331f77dae9d452184e8cd0cbf5f1b43: Received BeginTabletCopySession request for tablet f3b6691cbe8740d2978c20209c78f3ff from peer a29cd590999b4f32849ff5689cd53c11 ({username='slave'} at 127.21.84.136:34027)
I20240501 08:12:56.087111 27781 tablet_copy_service.cc:161] P 6331f77dae9d452184e8cd0cbf5f1b43: Beginning new tablet copy session on tablet f3b6691cbe8740d2978c20209c78f3ff from peer a29cd590999b4f32849ff5689cd53c11 at {username='slave'} at 127.21.84.136:34027: session id = a29cd590999b4f32849ff5689cd53c11-f3b6691cbe8740d2978c20209c78f3ff
I20240501 08:12:56.087204 3195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eb9763a1573455287b7137fe7fecf62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:56.090911 27781 tablet_copy_source_session.cc:205] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:56.092586 3168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3b6691cbe8740d2978c20209c78f3ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:56.096706 3195 tablet_copy_client.cc:783] T 8eb9763a1573455287b7137fe7fecf62 P 64998323a1f4482d8704bf20594d13a5: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:56.096958 3195 tablet_copy_client.cc:647] T 8eb9763a1573455287b7137fe7fecf62 P 64998323a1f4482d8704bf20594d13a5: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:56.099958 3195 tablet_copy_client.cc:523] T 8eb9763a1573455287b7137fe7fecf62 P 64998323a1f4482d8704bf20594d13a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:56.101537 3168 tablet_copy_client.cc:783] T f3b6691cbe8740d2978c20209c78f3ff P a29cd590999b4f32849ff5689cd53c11: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:56.101802 3168 tablet_copy_client.cc:647] T f3b6691cbe8740d2978c20209c78f3ff P a29cd590999b4f32849ff5689cd53c11: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:56.104118 3168 tablet_copy_client.cc:523] T f3b6691cbe8740d2978c20209c78f3ff P a29cd590999b4f32849ff5689cd53c11: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:56.106158 3195 tablet_bootstrap.cc:492] T 8eb9763a1573455287b7137fe7fecf62 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:12:56.107758 3168 tablet_bootstrap.cc:492] T f3b6691cbe8740d2978c20209c78f3ff P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:56.114521 3203 tablet_copy_client.cc:308] T a2981c0863bc41729605f8b076b6a6cb P a29cd590999b4f32849ff5689cd53c11: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.130:36503
I20240501 08:12:56.115392 27912 tablet_copy_service.cc:140] P f1525cbc82a24d13b65f136695c903af: Received BeginTabletCopySession request for tablet a2981c0863bc41729605f8b076b6a6cb from peer a29cd590999b4f32849ff5689cd53c11 ({username='slave'} at 127.21.84.136:54747)
I20240501 08:12:56.115547 27912 tablet_copy_service.cc:161] P f1525cbc82a24d13b65f136695c903af: Beginning new tablet copy session on tablet a2981c0863bc41729605f8b076b6a6cb from peer a29cd590999b4f32849ff5689cd53c11 at {username='slave'} at 127.21.84.136:54747: session id = a29cd590999b4f32849ff5689cd53c11-a2981c0863bc41729605f8b076b6a6cb
I20240501 08:12:56.119601 27912 tablet_copy_source_session.cc:205] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:56.121352 3203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2981c0863bc41729605f8b076b6a6cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:56.129879 3203 tablet_copy_client.cc:783] T a2981c0863bc41729605f8b076b6a6cb P a29cd590999b4f32849ff5689cd53c11: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:56.134308 3203 tablet_copy_client.cc:647] T a2981c0863bc41729605f8b076b6a6cb P a29cd590999b4f32849ff5689cd53c11: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:56.134660 2786 raft_consensus.cc:1050] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af: attempting to promote NON_VOTER 64998323a1f4482d8704bf20594d13a5 to VOTER
I20240501 08:12:56.135512 2786 consensus_queue.cc:237] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:12:56.141621 3203 tablet_copy_client.cc:523] T a2981c0863bc41729605f8b076b6a6cb P a29cd590999b4f32849ff5689cd53c11: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:56.146973 3203 tablet_bootstrap.cc:492] T a2981c0863bc41729605f8b076b6a6cb P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:12:56.163770 30801 raft_consensus.cc:1261] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:56.164755 27760 raft_consensus.cc:1261] T 728616fd583a4afea00e8fe52880e55b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:56.166128 2620 consensus_queue.cc:1035] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:56.167516 2620 consensus_queue.cc:1035] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:56.174234 28149 raft_consensus.cc:1261] T 728616fd583a4afea00e8fe52880e55b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:56.178763 3113 consensus_queue.cc:1035] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:56.189090 3113 raft_consensus.cc:2932] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:56.202386 27759 raft_consensus.cc:2932] T 728616fd583a4afea00e8fe52880e55b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:56.202428 30801 raft_consensus.cc:2932] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:56.210531 27758 raft_consensus.cc:1203] T e12867cd76504438abe91fbe15498230 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:56.213316 27618 catalog_manager.cc:5484] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:56.226756 27889 raft_consensus.cc:1203] T 80bb0e53796f4b319fa9f877451076f3 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:56.258818 28149 raft_consensus.cc:2932] T 728616fd583a4afea00e8fe52880e55b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:56.260746 3203 tablet_bootstrap.cc:492] T a2981c0863bc41729605f8b076b6a6cb P a29cd590999b4f32849ff5689cd53c11: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:56.261291 3203 tablet_bootstrap.cc:492] T a2981c0863bc41729605f8b076b6a6cb P a29cd590999b4f32849ff5689cd53c11: Bootstrap complete.
I20240501 08:12:56.261731 3203 ts_tablet_manager.cc:1367] T a2981c0863bc41729605f8b076b6a6cb P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.115s user 0.069s sys 0.008s
I20240501 08:12:56.290800 3203 raft_consensus.cc:348] T a2981c0863bc41729605f8b076b6a6cb P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:12:56.291357 3203 raft_consensus.cc:729] T a2981c0863bc41729605f8b076b6a6cb P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: LEARNER
I20240501 08:12:56.291693 3203 consensus_queue.cc:260] T a2981c0863bc41729605f8b076b6a6cb P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:12:56.294272 3203 ts_tablet_manager.cc:1398] T a2981c0863bc41729605f8b076b6a6cb P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.032s user 0.002s sys 0.000s
I20240501 08:12:56.295207 27912 tablet_copy_service.cc:342] P f1525cbc82a24d13b65f136695c903af: Request end of tablet copy session a29cd590999b4f32849ff5689cd53c11-a2981c0863bc41729605f8b076b6a6cb received from {username='slave'} at 127.21.84.136:54747
I20240501 08:12:56.295378 27912 tablet_copy_service.cc:433] P f1525cbc82a24d13b65f136695c903af: ending tablet copy session a29cd590999b4f32849ff5689cd53c11-a2981c0863bc41729605f8b076b6a6cb on tablet a2981c0863bc41729605f8b076b6a6cb with peer a29cd590999b4f32849ff5689cd53c11
W20240501 08:12:56.313023 27957 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.129:43471, user_credentials={real_user=slave}} blocked reactor thread for 111827us
I20240501 08:12:56.332731 30936 raft_consensus.cc:1203] T 24825c85238b4d618c65d00107135420 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240501 08:12:56.337586 27955 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.134:39933, user_credentials={real_user=slave}} blocked reactor thread for 145590us
I20240501 08:12:56.358680 3195 tablet_bootstrap.cc:492] T 8eb9763a1573455287b7137fe7fecf62 P 64998323a1f4482d8704bf20594d13a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:56.360987 3195 tablet_bootstrap.cc:492] T 8eb9763a1573455287b7137fe7fecf62 P 64998323a1f4482d8704bf20594d13a5: Bootstrap complete.
I20240501 08:12:56.361428 3195 ts_tablet_manager.cc:1367] T 8eb9763a1573455287b7137fe7fecf62 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.255s user 0.048s sys 0.006s
I20240501 08:12:56.362411 3195 raft_consensus.cc:348] T 8eb9763a1573455287b7137fe7fecf62 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
I20240501 08:12:56.362752 3195 raft_consensus.cc:729] T 8eb9763a1573455287b7137fe7fecf62 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: LEARNER
I20240501 08:12:56.363435 3195 consensus_queue.cc:260] T 8eb9763a1573455287b7137fe7fecf62 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
I20240501 08:12:56.364228 3195 ts_tablet_manager.cc:1398] T 8eb9763a1573455287b7137fe7fecf62 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:12:56.365595 28171 tablet_copy_service.cc:342] P 91d11eae5ef6432ca86eb8ed446b6ea6: Request end of tablet copy session 64998323a1f4482d8704bf20594d13a5-8eb9763a1573455287b7137fe7fecf62 received from {username='slave'} at 127.21.84.135:48917
I20240501 08:12:56.365713 28171 tablet_copy_service.cc:433] P 91d11eae5ef6432ca86eb8ed446b6ea6: ending tablet copy session 64998323a1f4482d8704bf20594d13a5-8eb9763a1573455287b7137fe7fecf62 on tablet 8eb9763a1573455287b7137fe7fecf62 with peer 64998323a1f4482d8704bf20594d13a5
I20240501 08:12:56.372175 3168 tablet_bootstrap.cc:492] T f3b6691cbe8740d2978c20209c78f3ff P a29cd590999b4f32849ff5689cd53c11: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:56.372663 3168 tablet_bootstrap.cc:492] T f3b6691cbe8740d2978c20209c78f3ff P a29cd590999b4f32849ff5689cd53c11: Bootstrap complete.
I20240501 08:12:56.372934 3168 ts_tablet_manager.cc:1367] T f3b6691cbe8740d2978c20209c78f3ff P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.265s user 0.117s sys 0.076s
I20240501 08:12:56.373697 3168 raft_consensus.cc:348] T f3b6691cbe8740d2978c20209c78f3ff P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:12:56.373991 3168 raft_consensus.cc:729] T f3b6691cbe8740d2978c20209c78f3ff P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: LEARNER
I20240501 08:12:56.374192 3168 consensus_queue.cc:260] T f3b6691cbe8740d2978c20209c78f3ff P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:12:56.374724 3168 ts_tablet_manager.cc:1398] T f3b6691cbe8740d2978c20209c78f3ff P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20240501 08:12:56.378957 30872 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.129:43471, user_credentials={real_user=slave}} blocked reactor thread for 190948us
I20240501 08:12:56.393007 27781 tablet_copy_service.cc:342] P 6331f77dae9d452184e8cd0cbf5f1b43: Request end of tablet copy session a29cd590999b4f32849ff5689cd53c11-f3b6691cbe8740d2978c20209c78f3ff received from {username='slave'} at 127.21.84.136:34027
I20240501 08:12:56.393157 27781 tablet_copy_service.cc:433] P 6331f77dae9d452184e8cd0cbf5f1b43: ending tablet copy session a29cd590999b4f32849ff5689cd53c11-f3b6691cbe8740d2978c20209c78f3ff on tablet f3b6691cbe8740d2978c20209c78f3ff with peer a29cd590999b4f32849ff5689cd53c11
I20240501 08:12:56.453500 30936 raft_consensus.cc:1203] T a2981c0863bc41729605f8b076b6a6cb P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:56.506222 27889 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "728616fd583a4afea00e8fe52880e55b"
dest_uuid: "f1525cbc82a24d13b65f136695c903af"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39922
I20240501 08:12:56.506487 27889 raft_consensus.cc:595] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Received request to transfer leadership
I20240501 08:12:56.535621 30537 consensus_queue.cc:237] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:12:56.540593 30936 raft_consensus.cc:1261] T 220add4033694ff381e023324a2a8b02 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:12:56.540717 30472 consensus_peers.cc:485] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 220add4033694ff381e023324a2a8b02. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:56.540843 27889 raft_consensus.cc:1261] T 220add4033694ff381e023324a2a8b02 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:56.541364 3188 consensus_queue.cc:1035] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:56.541764 3186 consensus_queue.cc:1035] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:56.545017 3188 raft_consensus.cc:2932] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:56.546267 27889 raft_consensus.cc:2932] T 220add4033694ff381e023324a2a8b02 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:56.546814 30936 raft_consensus.cc:2932] T 220add4033694ff381e023324a2a8b02 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:12:56.555888 27620 catalog_manager.cc:5484] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:56.591248 30936 raft_consensus.cc:1203] T f3b6691cbe8740d2978c20209c78f3ff P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:56.662079 2786 raft_consensus.cc:979] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af: : Instructing follower 6331f77dae9d452184e8cd0cbf5f1b43 to start an election
I20240501 08:12:56.662302 2620 raft_consensus.cc:1067] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Signalling peer 6331f77dae9d452184e8cd0cbf5f1b43 to start an election
I20240501 08:12:56.665446 27760 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "728616fd583a4afea00e8fe52880e55b"
dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
from {username='slave'} at 127.21.84.130:44301
I20240501 08:12:56.665676 27760 raft_consensus.cc:482] T 728616fd583a4afea00e8fe52880e55b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:56.665798 27760 raft_consensus.cc:3037] T 728616fd583a4afea00e8fe52880e55b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:56.669476 27760 raft_consensus.cc:504] T 728616fd583a4afea00e8fe52880e55b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:12:56.671661 28149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "728616fd583a4afea00e8fe52880e55b" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:12:56.671873 28149 raft_consensus.cc:3037] T 728616fd583a4afea00e8fe52880e55b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:56.672019 27760 leader_election.cc:288] T 728616fd583a4afea00e8fe52880e55b P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 2 election: Requested vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:56.671952 27889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "728616fd583a4afea00e8fe52880e55b" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:12:56.672178 27889 raft_consensus.cc:3032] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:12:56.672302 27889 raft_consensus.cc:729] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f1525cbc82a24d13b65f136695c903af, State: Running, Role: LEADER
I20240501 08:12:56.672674 27889 consensus_queue.cc:260] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:12:56.673228 27889 raft_consensus.cc:3037] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:56.674674 28149 raft_consensus.cc:2454] T 728616fd583a4afea00e8fe52880e55b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 2.
I20240501 08:12:56.676596 27889 raft_consensus.cc:2454] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 2.
I20240501 08:12:56.677563 27697 leader_election.cc:302] T 728616fd583a4afea00e8fe52880e55b P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:12:56.677551 30802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "728616fd583a4afea00e8fe52880e55b" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:12:56.677805 30802 raft_consensus.cc:3037] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:56.677917 3246 raft_consensus.cc:2781] T 728616fd583a4afea00e8fe52880e55b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:12:56.678073 3246 raft_consensus.cc:686] T 728616fd583a4afea00e8fe52880e55b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:12:56.678361 3246 consensus_queue.cc:237] T 728616fd583a4afea00e8fe52880e55b P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:12:56.680397 30802 raft_consensus.cc:2454] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 2.
I20240501 08:12:56.684311 27615 catalog_manager.cc:5484] T 728616fd583a4afea00e8fe52880e55b P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 1 to 2, leader changed from f1525cbc82a24d13b65f136695c903af (127.21.84.130) to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 2 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:56.685755 3188 raft_consensus.cc:1050] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6: attempting to promote NON_VOTER f1525cbc82a24d13b65f136695c903af to VOTER
I20240501 08:12:56.686655 3188 consensus_queue.cc:237] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:56.689662 27889 raft_consensus.cc:1261] T 80bb0e53796f4b319fa9f877451076f3 P f1525cbc82a24d13b65f136695c903af [term 1 LEARNER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:56.689911 30672 raft_consensus.cc:1261] T 80bb0e53796f4b319fa9f877451076f3 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:56.690017 28149 raft_consensus.cc:1261] T 80bb0e53796f4b319fa9f877451076f3 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:56.690954 3188 consensus_queue.cc:1035] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:56.691316 2594 consensus_queue.cc:1035] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:56.691897 3186 consensus_queue.cc:1035] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:56.733599 3186 raft_consensus.cc:2932] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:56.734838 30670 raft_consensus.cc:2932] T 80bb0e53796f4b319fa9f877451076f3 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:56.739182 27889 raft_consensus.cc:2932] T 80bb0e53796f4b319fa9f877451076f3 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:56.740666 28149 raft_consensus.cc:2932] T 80bb0e53796f4b319fa9f877451076f3 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:56.745494 27615 catalog_manager.cc:5484] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index 2 to 3, f1525cbc82a24d13b65f136695c903af (127.21.84.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:56.748256 2620 raft_consensus.cc:1050] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af: attempting to promote NON_VOTER a29cd590999b4f32849ff5689cd53c11 to VOTER
I20240501 08:12:56.749033 2620 consensus_queue.cc:237] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:12:56.751891 30537 consensus_queue.cc:237] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:12:56.754704 28149 raft_consensus.cc:1261] T 80bb0e53796f4b319fa9f877451076f3 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:56.755566 3188 consensus_queue.cc:1035] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:56.763665 27889 raft_consensus.cc:1261] T 80bb0e53796f4b319fa9f877451076f3 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:56.764227 3186 consensus_queue.cc:1035] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:56.767140 30936 raft_consensus.cc:1261] T 24825c85238b4d618c65d00107135420 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:56.769207 2786 consensus_queue.cc:1035] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:56.769726 2594 raft_consensus.cc:2932] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:56.770939 28020 raft_consensus.cc:1261] T 24825c85238b4d618c65d00107135420 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:56.771612 27760 raft_consensus.cc:1261] T 24825c85238b4d618c65d00107135420 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:56.771539 2786 consensus_queue.cc:1035] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:56.774540 2786 consensus_queue.cc:1035] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:56.776094 27607 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 80bb0e53796f4b319fa9f877451076f3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:56.779070 30802 raft_consensus.cc:1203] T 8eb9763a1573455287b7137fe7fecf62 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:56.781276 27889 raft_consensus.cc:2932] T 80bb0e53796f4b319fa9f877451076f3 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:56.788317 27618 catalog_manager.cc:5484] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:56.774242 28149 raft_consensus.cc:2932] T 80bb0e53796f4b319fa9f877451076f3 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } }
I20240501 08:12:56.795661 30650 tablet_service.cc:1514] Processing DeleteTablet for tablet 80bb0e53796f4b319fa9f877451076f3 with delete_type TABLET_DATA_TOMBSTONED (TS 9a022e1a501f4d44a0468054d6c3de47 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57156
I20240501 08:12:56.798555 3312 tablet_replica.cc:330] T 80bb0e53796f4b319fa9f877451076f3 P 9a022e1a501f4d44a0468054d6c3de47: stopping tablet replica
I20240501 08:12:56.798915 3312 raft_consensus.cc:2229] T 80bb0e53796f4b319fa9f877451076f3 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:56.799283 3312 raft_consensus.cc:2258] T 80bb0e53796f4b319fa9f877451076f3 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:56.800637 3297 raft_consensus.cc:1050] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6: attempting to promote NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 to VOTER
I20240501 08:12:56.801843 3312 ts_tablet_manager.cc:1875] T 80bb0e53796f4b319fa9f877451076f3 P 9a022e1a501f4d44a0468054d6c3de47: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:56.801832 3297 consensus_queue.cc:237] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:12:56.809351 27889 raft_consensus.cc:1261] T e12867cd76504438abe91fbe15498230 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:56.810086 3188 consensus_queue.cc:1035] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:56.810169 27758 raft_consensus.cc:1261] T e12867cd76504438abe91fbe15498230 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:56.810575 3312 ts_tablet_manager.cc:1888] T 80bb0e53796f4b319fa9f877451076f3 P 9a022e1a501f4d44a0468054d6c3de47: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:56.810748 3312 log.cc:1196] T 80bb0e53796f4b319fa9f877451076f3 P 9a022e1a501f4d44a0468054d6c3de47: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/wals/80bb0e53796f4b319fa9f877451076f3
I20240501 08:12:56.811333 3188 consensus_queue.cc:1035] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:56.811726 27604 catalog_manager.cc:4830] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): tablet 80bb0e53796f4b319fa9f877451076f3 (table rebalancer_extra_table_50 [id=95faa655da1346f68f96519e9154b964]) successfully deleted
I20240501 08:12:56.817663 30670 raft_consensus.cc:1261] T e12867cd76504438abe91fbe15498230 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:56.818778 2594 consensus_queue.cc:1035] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:56.834200 3226 raft_consensus.cc:2932] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:56.844110 27760 raft_consensus.cc:2932] T 24825c85238b4d618c65d00107135420 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:56.847352 2594 raft_consensus.cc:2932] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:56.855170 28020 raft_consensus.cc:2932] T 24825c85238b4d618c65d00107135420 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:56.861135 27891 raft_consensus.cc:2932] T e12867cd76504438abe91fbe15498230 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:56.870067 27619 catalog_manager.cc:5484] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af reported cstate change: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:56.884687 30670 raft_consensus.cc:2932] T e12867cd76504438abe91fbe15498230 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:56.927631 27619 catalog_manager.cc:5484] T e12867cd76504438abe91fbe15498230 P f1525cbc82a24d13b65f136695c903af reported cstate change: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:56.931053 30934 raft_consensus.cc:2932] T 24825c85238b4d618c65d00107135420 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:56.940691 2786 raft_consensus.cc:1050] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af: attempting to promote NON_VOTER a29cd590999b4f32849ff5689cd53c11 to VOTER
I20240501 08:12:56.942025 2786 consensus_queue.cc:237] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:12:56.947901 30936 raft_consensus.cc:1261] T a2981c0863bc41729605f8b076b6a6cb P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:56.948079 28149 raft_consensus.cc:1261] T a2981c0863bc41729605f8b076b6a6cb P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:56.948278 28020 raft_consensus.cc:1261] T a2981c0863bc41729605f8b076b6a6cb P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:56.948410 2620 consensus_queue.cc:1035] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:56.948726 3319 consensus_queue.cc:1035] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:56.949525 3226 consensus_queue.cc:1035] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:56.969503 2620 raft_consensus.cc:2932] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:56.974242 30934 raft_consensus.cc:2932] T a2981c0863bc41729605f8b076b6a6cb P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:56.983381 28020 raft_consensus.cc:2932] T a2981c0863bc41729605f8b076b6a6cb P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:57.001297 27612 catalog_manager.cc:5484] T a2981c0863bc41729605f8b076b6a6cb P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:57.007762 28151 raft_consensus.cc:2932] T a2981c0863bc41729605f8b076b6a6cb P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:57.048363 27760 raft_consensus.cc:2932] T e12867cd76504438abe91fbe15498230 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:57.060292 30536 consensus_queue.cc:237] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:12:57.062820 30672 raft_consensus.cc:1261] T e12867cd76504438abe91fbe15498230 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:57.067288 2594 consensus_queue.cc:1035] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:57.070289 2594 raft_consensus.cc:2932] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:57.070736 30672 raft_consensus.cc:2932] T e12867cd76504438abe91fbe15498230 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:57.073176 27889 consensus_queue.cc:237] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:12:57.076366 27607 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e12867cd76504438abe91fbe15498230 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:57.076881 30934 raft_consensus.cc:1261] T a2981c0863bc41729605f8b076b6a6cb P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:57.077114 27891 consensus_queue.cc:237] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:12:57.078091 28149 raft_consensus.cc:1261] T a2981c0863bc41729605f8b076b6a6cb P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:57.078675 3226 consensus_queue.cc:1035] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:57.079685 3246 raft_consensus.cc:1050] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43: attempting to promote NON_VOTER a29cd590999b4f32849ff5689cd53c11 to VOTER
I20240501 08:12:57.080487 3246 consensus_queue.cc:237] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:12:57.082886 2620 consensus_queue.cc:1035] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:57.085265 27757 raft_consensus.cc:1261] T 24825c85238b4d618c65d00107135420 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:57.086002 30934 raft_consensus.cc:1261] T 24825c85238b4d618c65d00107135420 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:57.085786 2786 raft_consensus.cc:2932] T a2981c0863bc41729605f8b076b6a6cb P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:57.087427 27759 raft_consensus.cc:1261] T e12867cd76504438abe91fbe15498230 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:57.088598 27618 catalog_manager.cc:5484] T e12867cd76504438abe91fbe15498230 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 3 to 4, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:57.092191 3226 consensus_queue.cc:1035] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:57.093757 3188 consensus_queue.cc:1035] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:57.094496 28149 raft_consensus.cc:2932] T a2981c0863bc41729605f8b076b6a6cb P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:57.096185 2620 consensus_queue.cc:1035] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:57.096242 27760 raft_consensus.cc:2932] T e12867cd76504438abe91fbe15498230 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:57.098266 30934 raft_consensus.cc:2932] T a2981c0863bc41729605f8b076b6a6cb P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:57.100158 2620 raft_consensus.cc:2932] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:57.101784 3327 ts_tablet_manager.cc:902] T 220add4033694ff381e023324a2a8b02 P 6331f77dae9d452184e8cd0cbf5f1b43: Initiating tablet copy from peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:12:57.107234 27615 catalog_manager.cc:5484] T a2981c0863bc41729605f8b076b6a6cb P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:57.108157 27607 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 24825c85238b4d618c65d00107135420 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:57.108609 27871 tablet_service.cc:1514] Processing DeleteTablet for tablet e12867cd76504438abe91fbe15498230 with delete_type TABLET_DATA_TOMBSTONED (TS f1525cbc82a24d13b65f136695c903af not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:57.111539 28149 raft_consensus.cc:1261] T f3b6691cbe8740d2978c20209c78f3ff P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:57.112751 28022 raft_consensus.cc:1261] T f3b6691cbe8740d2978c20209c78f3ff P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:57.113585 2674 consensus_queue.cc:1035] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:57.114439 30936 raft_consensus.cc:1261] T f3b6691cbe8740d2978c20209c78f3ff P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:57.116226 27607 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a2981c0863bc41729605f8b076b6a6cb with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:57.116292 2674 consensus_queue.cc:1035] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:57.117589 27761 raft_consensus.cc:2932] T 24825c85238b4d618c65d00107135420 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:57.127614 2674 raft_consensus.cc:2932] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:57.129074 3331 tablet_replica.cc:330] T e12867cd76504438abe91fbe15498230 P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:57.129279 3331 raft_consensus.cc:2229] T e12867cd76504438abe91fbe15498230 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:57.131232 28000 tablet_service.cc:1514] Processing DeleteTablet for tablet a2981c0863bc41729605f8b076b6a6cb with delete_type TABLET_DATA_TOMBSTONED (TS aad81047009a4ffcbc450b60030583b5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:57.134238 3331 raft_consensus.cc:2258] T e12867cd76504438abe91fbe15498230 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:57.137467 3331 ts_tablet_manager.cc:1875] T e12867cd76504438abe91fbe15498230 P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:57.139102 30936 raft_consensus.cc:2932] T f3b6691cbe8740d2978c20209c78f3ff P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:57.110816 30934 raft_consensus.cc:2932] T 24825c85238b4d618c65d00107135420 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:57.142517 2933 consensus_queue.cc:1035] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:57.144644 28022 raft_consensus.cc:2932] T f3b6691cbe8740d2978c20209c78f3ff P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:57.145898 27618 catalog_manager.cc:5484] T 24825c85238b4d618c65d00107135420 P f1525cbc82a24d13b65f136695c903af reported cstate change: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:57.151360 27620 catalog_manager.cc:5484] T f3b6691cbe8740d2978c20209c78f3ff P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:57.155426 3331 ts_tablet_manager.cc:1888] T e12867cd76504438abe91fbe15498230 P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:57.161190 28002 tablet_service.cc:1514] Processing DeleteTablet for tablet 24825c85238b4d618c65d00107135420 with delete_type TABLET_DATA_TOMBSTONED (TS aad81047009a4ffcbc450b60030583b5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:57.166266 3331 log.cc:1196] T e12867cd76504438abe91fbe15498230 P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/e12867cd76504438abe91fbe15498230
I20240501 08:12:57.171167 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet e12867cd76504438abe91fbe15498230 (table rebalancer_extra_table_26 [id=444d193c9c4c4c16969e6c88e6e590c9]) successfully deleted
I20240501 08:12:57.179685 30801 raft_consensus.cc:1261] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:12:57.180346 3246 consensus_queue.cc:1035] T 728616fd583a4afea00e8fe52880e55b P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:57.181552 3336 tablet_replica.cc:330] T a2981c0863bc41729605f8b076b6a6cb P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:57.183307 3336 raft_consensus.cc:2229] T a2981c0863bc41729605f8b076b6a6cb P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:57.184666 28151 raft_consensus.cc:2932] T f3b6691cbe8740d2978c20209c78f3ff P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:57.190850 3336 raft_consensus.cc:2258] T a2981c0863bc41729605f8b076b6a6cb P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:57.196211 3336 ts_tablet_manager.cc:1875] T a2981c0863bc41729605f8b076b6a6cb P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:57.201661 3327 tablet_copy_client.cc:308] T 220add4033694ff381e023324a2a8b02 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.133:38293
I20240501 08:12:57.202404 30559 tablet_copy_service.cc:140] P 31811708c6774037b82df44024b229b6: Received BeginTabletCopySession request for tablet 220add4033694ff381e023324a2a8b02 from peer 6331f77dae9d452184e8cd0cbf5f1b43 ({username='slave'} at 127.21.84.129:35413)
I20240501 08:12:57.202575 30559 tablet_copy_service.cc:161] P 31811708c6774037b82df44024b229b6: Beginning new tablet copy session on tablet 220add4033694ff381e023324a2a8b02 from peer 6331f77dae9d452184e8cd0cbf5f1b43 at {username='slave'} at 127.21.84.129:35413: session id = 6331f77dae9d452184e8cd0cbf5f1b43-220add4033694ff381e023324a2a8b02
I20240501 08:12:57.204994 30559 tablet_copy_source_session.cc:205] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:57.206040 3336 ts_tablet_manager.cc:1888] T a2981c0863bc41729605f8b076b6a6cb P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:57.206168 3336 log.cc:1196] T a2981c0863bc41729605f8b076b6a6cb P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/a2981c0863bc41729605f8b076b6a6cb
I20240501 08:12:57.206247 3327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 220add4033694ff381e023324a2a8b02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:57.206885 3336 tablet_replica.cc:330] T 24825c85238b4d618c65d00107135420 P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:57.206933 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet a2981c0863bc41729605f8b076b6a6cb (table rebalance_test_table_9 [id=e6efea2767a741beae5c93339e7712ba]) successfully deleted
I20240501 08:12:57.207273 3336 raft_consensus.cc:2229] T 24825c85238b4d618c65d00107135420 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:57.207656 3336 raft_consensus.cc:2258] T 24825c85238b4d618c65d00107135420 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:57.208636 3336 ts_tablet_manager.cc:1875] T 24825c85238b4d618c65d00107135420 P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:57.211786 3327 tablet_copy_client.cc:783] T 220add4033694ff381e023324a2a8b02 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:57.212049 3327 tablet_copy_client.cc:647] T 220add4033694ff381e023324a2a8b02 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:57.212811 28151 raft_consensus.cc:1261] T 728616fd583a4afea00e8fe52880e55b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:12:57.213311 3332 consensus_queue.cc:1035] T 728616fd583a4afea00e8fe52880e55b P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:57.213496 3327 tablet_copy_client.cc:523] T 220add4033694ff381e023324a2a8b02 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:57.216722 3327 tablet_bootstrap.cc:492] T 220add4033694ff381e023324a2a8b02 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:12:57.221108 3336 ts_tablet_manager.cc:1888] T 24825c85238b4d618c65d00107135420 P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:57.222054 3336 log.cc:1196] T 24825c85238b4d618c65d00107135420 P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/24825c85238b4d618c65d00107135420
I20240501 08:12:57.223089 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet 24825c85238b4d618c65d00107135420 (table rebalance_test_table_19 [id=4fc7c32c4ee342e5804c1a70b815d952]) successfully deleted
I20240501 08:12:57.228401 27889 raft_consensus.cc:1261] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [term 2 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:12:57.228958 3327 tablet_bootstrap.cc:492] T 220add4033694ff381e023324a2a8b02 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:57.229312 3327 tablet_bootstrap.cc:492] T 220add4033694ff381e023324a2a8b02 P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap complete.
I20240501 08:12:57.229701 3327 ts_tablet_manager.cc:1367] T 220add4033694ff381e023324a2a8b02 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.007s
I20240501 08:12:57.229615 3246 consensus_queue.cc:1035] T 728616fd583a4afea00e8fe52880e55b P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:57.230674 3327 raft_consensus.cc:348] T 220add4033694ff381e023324a2a8b02 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:12:57.231104 3327 raft_consensus.cc:729] T 220add4033694ff381e023324a2a8b02 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: LEARNER
I20240501 08:12:57.231401 3327 consensus_queue.cc:260] T 220add4033694ff381e023324a2a8b02 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:12:57.232125 3327 ts_tablet_manager.cc:1398] T 220add4033694ff381e023324a2a8b02 P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:57.235116 30559 tablet_copy_service.cc:342] P 31811708c6774037b82df44024b229b6: Request end of tablet copy session 6331f77dae9d452184e8cd0cbf5f1b43-220add4033694ff381e023324a2a8b02 received from {username='slave'} at 127.21.84.129:35413
I20240501 08:12:57.235342 30559 tablet_copy_service.cc:433] P 31811708c6774037b82df44024b229b6: ending tablet copy session 6331f77dae9d452184e8cd0cbf5f1b43-220add4033694ff381e023324a2a8b02 on tablet 220add4033694ff381e023324a2a8b02 with peer 6331f77dae9d452184e8cd0cbf5f1b43
I20240501 08:12:57.254909 27761 consensus_queue.cc:237] T 728616fd583a4afea00e8fe52880e55b P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:12:57.257217 28151 raft_consensus.cc:1261] T 728616fd583a4afea00e8fe52880e55b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:12:57.258095 3332 consensus_queue.cc:1035] T 728616fd583a4afea00e8fe52880e55b P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:57.259675 30801 raft_consensus.cc:1261] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:12:57.260192 3332 consensus_queue.cc:1035] T 728616fd583a4afea00e8fe52880e55b P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:12:57.260768 3332 raft_consensus.cc:2932] T 728616fd583a4afea00e8fe52880e55b P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:57.261389 28151 raft_consensus.cc:2932] T 728616fd583a4afea00e8fe52880e55b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:57.265381 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 728616fd583a4afea00e8fe52880e55b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:57.267532 27619 catalog_manager.cc:5484] T 728616fd583a4afea00e8fe52880e55b P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: config changed from index 5 to 7, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New cstate: current_term: 2 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:57.271479 27871 tablet_service.cc:1514] Processing DeleteTablet for tablet 728616fd583a4afea00e8fe52880e55b with delete_type TABLET_DATA_TOMBSTONED (TS f1525cbc82a24d13b65f136695c903af not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:57.271826 3331 tablet_replica.cc:330] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:57.272034 3331 raft_consensus.cc:2229] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:57.272210 3331 raft_consensus.cc:2258] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:57.272970 3331 ts_tablet_manager.cc:1875] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:57.280208 3331 ts_tablet_manager.cc:1888] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 08:12:57.280339 3331 log.cc:1196] T 728616fd583a4afea00e8fe52880e55b P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/728616fd583a4afea00e8fe52880e55b
I20240501 08:12:57.281136 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet 728616fd583a4afea00e8fe52880e55b (table rebalance_test_table_7 [id=1ae463b433bd4ed6bb9ab1bdc10dfdf6]) successfully deleted
I20240501 08:12:57.282351 30801 raft_consensus.cc:2932] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:57.314612 2222 raft_consensus.cc:1050] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6: attempting to promote NON_VOTER 64998323a1f4482d8704bf20594d13a5 to VOTER
I20240501 08:12:57.315431 2222 consensus_queue.cc:237] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:12:57.334596 30801 raft_consensus.cc:1261] T 8eb9763a1573455287b7137fe7fecf62 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:57.335702 27889 raft_consensus.cc:1261] T 8eb9763a1573455287b7137fe7fecf62 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:57.338518 28022 raft_consensus.cc:1261] T 8eb9763a1573455287b7137fe7fecf62 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:57.339112 3016 consensus_queue.cc:1035] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:57.342206 2222 consensus_queue.cc:1035] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:57.342550 2607 consensus_queue.cc:1035] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:57.355263 2607 raft_consensus.cc:2932] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:57.356127 28020 raft_consensus.cc:2932] T 8eb9763a1573455287b7137fe7fecf62 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:57.362754 30801 raft_consensus.cc:2932] T 8eb9763a1573455287b7137fe7fecf62 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:57.364305 27619 catalog_manager.cc:5484] T 8eb9763a1573455287b7137fe7fecf62 P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:57.364300 27889 raft_consensus.cc:2932] T 8eb9763a1573455287b7137fe7fecf62 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:57.438463 28151 consensus_queue.cc:237] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:12:57.444197 30802 raft_consensus.cc:1261] T 8eb9763a1573455287b7137fe7fecf62 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:57.444824 3359 consensus_queue.cc:1035] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:57.445231 28022 raft_consensus.cc:1261] T 8eb9763a1573455287b7137fe7fecf62 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:57.446127 3354 consensus_queue.cc:1035] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:57.447759 3354 raft_consensus.cc:2932] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:57.450256 30802 raft_consensus.cc:2932] T 8eb9763a1573455287b7137fe7fecf62 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:57.452287 27604 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8eb9763a1573455287b7137fe7fecf62 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:57.459034 27620 catalog_manager.cc:5484] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: config changed from index 5 to 6, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:57.463729 27871 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eb9763a1573455287b7137fe7fecf62 with delete_type TABLET_DATA_TOMBSTONED (TS f1525cbc82a24d13b65f136695c903af not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:57.464083 3331 tablet_replica.cc:330] T 8eb9763a1573455287b7137fe7fecf62 P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:57.464304 3331 raft_consensus.cc:2229] T 8eb9763a1573455287b7137fe7fecf62 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:57.464529 3331 raft_consensus.cc:2258] T 8eb9763a1573455287b7137fe7fecf62 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:57.465567 3331 ts_tablet_manager.cc:1875] T 8eb9763a1573455287b7137fe7fecf62 P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:57.473416 28022 raft_consensus.cc:2932] T 8eb9763a1573455287b7137fe7fecf62 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:12:57.475231 3331 ts_tablet_manager.cc:1888] T 8eb9763a1573455287b7137fe7fecf62 P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:57.475421 3331 log.cc:1196] T 8eb9763a1573455287b7137fe7fecf62 P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/8eb9763a1573455287b7137fe7fecf62
I20240501 08:12:57.476135 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet 8eb9763a1573455287b7137fe7fecf62 (table rebalance_test_table_19 [id=4fc7c32c4ee342e5804c1a70b815d952]) successfully deleted
I20240501 08:12:57.628226 28020 consensus_queue.cc:237] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:12:57.630844 27891 raft_consensus.cc:1261] T 52b69ecbe4094f32a1ed73b71b934451 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:57.631407 3237 consensus_queue.cc:1035] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:57.631626 28152 raft_consensus.cc:1261] T 52b69ecbe4094f32a1ed73b71b934451 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:12:57.634069 27955 consensus_peers.cc:485] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 -> Peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Couldn't send request to peer 9a022e1a501f4d44a0468054d6c3de47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 52b69ecbe4094f32a1ed73b71b934451. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:57.634361 3237 consensus_queue.cc:1035] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:57.638232 3134 raft_consensus.cc:2932] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:57.638978 27891 raft_consensus.cc:2932] T 52b69ecbe4094f32a1ed73b71b934451 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:57.643860 27760 raft_consensus.cc:1203] T 220add4033694ff381e023324a2a8b02 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:57.645247 27615 catalog_manager.cc:5484] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index -1 to 4, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:57.653992 28152 raft_consensus.cc:2932] T 52b69ecbe4094f32a1ed73b71b934451 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:12:57.680547 30801 consensus_queue.cc:237] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:12:57.683656 27891 raft_consensus.cc:1261] T bc84d1d413994a76bd6cf42335438755 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:57.684407 30537 raft_consensus.cc:1261] T bc84d1d413994a76bd6cf42335438755 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:57.684962 3413 consensus_queue.cc:1035] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:57.685377 3109 consensus_queue.cc:1035] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:12:57.685771 30739 consensus_peers.cc:485] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 -> Peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Couldn't send request to peer aad81047009a4ffcbc450b60030583b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bc84d1d413994a76bd6cf42335438755. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:57.687220 27760 consensus_queue.cc:237] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:12:57.688920 30934 raft_consensus.cc:1261] T f3b6691cbe8740d2978c20209c78f3ff P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:57.689064 28151 raft_consensus.cc:1261] T f3b6691cbe8740d2978c20209c78f3ff P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:57.689742 3246 consensus_queue.cc:1035] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:57.690014 2933 consensus_queue.cc:1035] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:57.692628 3246 raft_consensus.cc:2932] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:57.694525 28151 raft_consensus.cc:2932] T f3b6691cbe8740d2978c20209c78f3ff P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:57.694530 3109 raft_consensus.cc:2932] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:57.697198 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f3b6691cbe8740d2978c20209c78f3ff with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:57.697000 30537 raft_consensus.cc:2932] T bc84d1d413994a76bd6cf42335438755 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:57.703819 27615 catalog_manager.cc:5484] T bc84d1d413994a76bd6cf42335438755 P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:57.708477 30934 raft_consensus.cc:2932] T f3b6691cbe8740d2978c20209c78f3ff P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:12:57.709945 27620 catalog_manager.cc:5484] T f3b6691cbe8740d2978c20209c78f3ff P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index 5 to 6, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:57.717905 27891 raft_consensus.cc:2932] T bc84d1d413994a76bd6cf42335438755 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:12:57.723438 28000 tablet_service.cc:1514] Processing DeleteTablet for tablet f3b6691cbe8740d2978c20209c78f3ff with delete_type TABLET_DATA_TOMBSTONED (TS aad81047009a4ffcbc450b60030583b5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41320
I20240501 08:12:57.723786 3336 tablet_replica.cc:330] T f3b6691cbe8740d2978c20209c78f3ff P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:12:57.723984 3336 raft_consensus.cc:2229] T f3b6691cbe8740d2978c20209c78f3ff P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:57.724274 3336 raft_consensus.cc:2258] T f3b6691cbe8740d2978c20209c78f3ff P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:57.725317 3336 ts_tablet_manager.cc:1875] T f3b6691cbe8740d2978c20209c78f3ff P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:57.732762 3336 ts_tablet_manager.cc:1888] T f3b6691cbe8740d2978c20209c78f3ff P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:57.732899 3336 log.cc:1196] T f3b6691cbe8740d2978c20209c78f3ff P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/f3b6691cbe8740d2978c20209c78f3ff
I20240501 08:12:57.733564 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet f3b6691cbe8740d2978c20209c78f3ff (table rebalance_test_table_13 [id=86d1b715252040a3bc0f714b3db3f4ff]) successfully deleted
I20240501 08:12:58.009617 3452 ts_tablet_manager.cc:902] T 52b69ecbe4094f32a1ed73b71b934451 P 9a022e1a501f4d44a0468054d6c3de47: Initiating tablet copy from peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:12:58.010835 3452 tablet_copy_client.cc:308] T 52b69ecbe4094f32a1ed73b71b934451 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.131:36305
I20240501 08:12:58.011457 28042 tablet_copy_service.cc:140] P aad81047009a4ffcbc450b60030583b5: Received BeginTabletCopySession request for tablet 52b69ecbe4094f32a1ed73b71b934451 from peer 9a022e1a501f4d44a0468054d6c3de47 ({username='slave'} at 127.21.84.134:40301)
I20240501 08:12:58.011595 28042 tablet_copy_service.cc:161] P aad81047009a4ffcbc450b60030583b5: Beginning new tablet copy session on tablet 52b69ecbe4094f32a1ed73b71b934451 from peer 9a022e1a501f4d44a0468054d6c3de47 at {username='slave'} at 127.21.84.134:40301: session id = 9a022e1a501f4d44a0468054d6c3de47-52b69ecbe4094f32a1ed73b71b934451
I20240501 08:12:58.013490 28042 tablet_copy_source_session.cc:205] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:58.014726 3452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52b69ecbe4094f32a1ed73b71b934451. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:58.020144 3452 tablet_copy_client.cc:783] T 52b69ecbe4094f32a1ed73b71b934451 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:58.020349 3452 tablet_copy_client.cc:647] T 52b69ecbe4094f32a1ed73b71b934451 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:58.022116 3452 tablet_copy_client.cc:523] T 52b69ecbe4094f32a1ed73b71b934451 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:58.025432 3452 tablet_bootstrap.cc:492] T 52b69ecbe4094f32a1ed73b71b934451 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:12:58.060653 3452 tablet_bootstrap.cc:492] T 52b69ecbe4094f32a1ed73b71b934451 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:58.061156 3452 tablet_bootstrap.cc:492] T 52b69ecbe4094f32a1ed73b71b934451 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap complete.
I20240501 08:12:58.061416 3452 ts_tablet_manager.cc:1367] T 52b69ecbe4094f32a1ed73b71b934451 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.036s user 0.034s sys 0.004s
I20240501 08:12:58.062333 3452 raft_consensus.cc:348] T 52b69ecbe4094f32a1ed73b71b934451 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:12:58.062662 3452 raft_consensus.cc:729] T 52b69ecbe4094f32a1ed73b71b934451 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: LEARNER
I20240501 08:12:58.062870 3452 consensus_queue.cc:260] T 52b69ecbe4094f32a1ed73b71b934451 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:12:58.063582 3452 ts_tablet_manager.cc:1398] T 52b69ecbe4094f32a1ed73b71b934451 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:58.064682 28042 tablet_copy_service.cc:342] P aad81047009a4ffcbc450b60030583b5: Request end of tablet copy session 9a022e1a501f4d44a0468054d6c3de47-52b69ecbe4094f32a1ed73b71b934451 received from {username='slave'} at 127.21.84.134:40301
I20240501 08:12:58.064921 28042 tablet_copy_service.cc:433] P aad81047009a4ffcbc450b60030583b5: ending tablet copy session 9a022e1a501f4d44a0468054d6c3de47-52b69ecbe4094f32a1ed73b71b934451 on tablet 52b69ecbe4094f32a1ed73b71b934451 with peer 9a022e1a501f4d44a0468054d6c3de47
I20240501 08:12:58.074791 3460 ts_tablet_manager.cc:902] T bc84d1d413994a76bd6cf42335438755 P aad81047009a4ffcbc450b60030583b5: Initiating tablet copy from peer 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:12:58.076174 3460 tablet_copy_client.cc:308] T bc84d1d413994a76bd6cf42335438755 P aad81047009a4ffcbc450b60030583b5: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.135:33503
I20240501 08:12:58.076915 30825 tablet_copy_service.cc:140] P 64998323a1f4482d8704bf20594d13a5: Received BeginTabletCopySession request for tablet bc84d1d413994a76bd6cf42335438755 from peer aad81047009a4ffcbc450b60030583b5 ({username='slave'} at 127.21.84.131:43681)
I20240501 08:12:58.077056 30825 tablet_copy_service.cc:161] P 64998323a1f4482d8704bf20594d13a5: Beginning new tablet copy session on tablet bc84d1d413994a76bd6cf42335438755 from peer aad81047009a4ffcbc450b60030583b5 at {username='slave'} at 127.21.84.131:43681: session id = aad81047009a4ffcbc450b60030583b5-bc84d1d413994a76bd6cf42335438755
I20240501 08:12:58.078480 3320 raft_consensus.cc:1050] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6: attempting to promote NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 to VOTER
I20240501 08:12:58.079299 3320 consensus_queue.cc:237] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:12:58.079725 30825 tablet_copy_source_session.cc:205] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:58.080909 3460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc84d1d413994a76bd6cf42335438755. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:58.083379 27891 raft_consensus.cc:1261] T 220add4033694ff381e023324a2a8b02 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:58.083671 27760 raft_consensus.cc:1261] T 220add4033694ff381e023324a2a8b02 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:58.083976 2594 consensus_queue.cc:1035] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:58.085565 2594 consensus_queue.cc:1035] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:58.086838 3460 tablet_copy_client.cc:783] T bc84d1d413994a76bd6cf42335438755 P aad81047009a4ffcbc450b60030583b5: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:58.087098 3460 tablet_copy_client.cc:647] T bc84d1d413994a76bd6cf42335438755 P aad81047009a4ffcbc450b60030583b5: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:58.088799 3460 tablet_copy_client.cc:523] T bc84d1d413994a76bd6cf42335438755 P aad81047009a4ffcbc450b60030583b5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:58.093151 3460 tablet_bootstrap.cc:492] T bc84d1d413994a76bd6cf42335438755 P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:12:58.099958 3460 tablet_bootstrap.cc:492] T bc84d1d413994a76bd6cf42335438755 P aad81047009a4ffcbc450b60030583b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:58.100394 3460 tablet_bootstrap.cc:492] T bc84d1d413994a76bd6cf42335438755 P aad81047009a4ffcbc450b60030583b5: Bootstrap complete.
I20240501 08:12:58.100556 30934 raft_consensus.cc:1261] T 220add4033694ff381e023324a2a8b02 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:58.100243 2594 raft_consensus.cc:2932] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:58.100730 3460 ts_tablet_manager.cc:1367] T bc84d1d413994a76bd6cf42335438755 P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.008s user 0.010s sys 0.000s
I20240501 08:12:58.101397 3460 raft_consensus.cc:348] T bc84d1d413994a76bd6cf42335438755 P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:12:58.101639 3460 raft_consensus.cc:729] T bc84d1d413994a76bd6cf42335438755 P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: LEARNER
I20240501 08:12:58.101824 3460 consensus_queue.cc:260] T bc84d1d413994a76bd6cf42335438755 P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:12:58.102807 3460 ts_tablet_manager.cc:1398] T bc84d1d413994a76bd6cf42335438755 P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:58.105742 3469 consensus_queue.cc:1035] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:58.109090 30825 tablet_copy_service.cc:342] P 64998323a1f4482d8704bf20594d13a5: Request end of tablet copy session aad81047009a4ffcbc450b60030583b5-bc84d1d413994a76bd6cf42335438755 received from {username='slave'} at 127.21.84.131:43681
I20240501 08:12:58.109216 30825 tablet_copy_service.cc:433] P 64998323a1f4482d8704bf20594d13a5: ending tablet copy session aad81047009a4ffcbc450b60030583b5-bc84d1d413994a76bd6cf42335438755 on tablet bc84d1d413994a76bd6cf42335438755 with peer aad81047009a4ffcbc450b60030583b5
I20240501 08:12:58.131378 30936 raft_consensus.cc:2932] T 220add4033694ff381e023324a2a8b02 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:58.131486 27760 raft_consensus.cc:2932] T 220add4033694ff381e023324a2a8b02 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:58.139720 27620 catalog_manager.cc:5484] T 220add4033694ff381e023324a2a8b02 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:58.151955 27891 raft_consensus.cc:2932] T 220add4033694ff381e023324a2a8b02 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:58.162309 30537 consensus_queue.cc:237] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:12:58.165681 27760 raft_consensus.cc:1261] T 220add4033694ff381e023324a2a8b02 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:58.166209 30934 raft_consensus.cc:1261] T 220add4033694ff381e023324a2a8b02 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:58.166848 3473 consensus_queue.cc:1035] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:58.168891 3469 consensus_queue.cc:1035] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:58.171770 2594 raft_consensus.cc:2932] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:58.172989 30934 raft_consensus.cc:2932] T 220add4033694ff381e023324a2a8b02 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:58.175559 27607 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 220add4033694ff381e023324a2a8b02 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:58.179287 27618 catalog_manager.cc:5484] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index 3 to 4, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:58.181283 27760 raft_consensus.cc:2932] T 220add4033694ff381e023324a2a8b02 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:12:58.184640 27871 tablet_service.cc:1514] Processing DeleteTablet for tablet 220add4033694ff381e023324a2a8b02 with delete_type TABLET_DATA_TOMBSTONED (TS f1525cbc82a24d13b65f136695c903af not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:58.194849 3481 tablet_replica.cc:330] T 220add4033694ff381e023324a2a8b02 P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:58.195158 3481 raft_consensus.cc:2229] T 220add4033694ff381e023324a2a8b02 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:58.195462 3481 raft_consensus.cc:2258] T 220add4033694ff381e023324a2a8b02 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:58.196336 3481 ts_tablet_manager.cc:1875] T 220add4033694ff381e023324a2a8b02 P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:58.201279 3481 ts_tablet_manager.cc:1888] T 220add4033694ff381e023324a2a8b02 P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:58.201474 3481 log.cc:1196] T 220add4033694ff381e023324a2a8b02 P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/220add4033694ff381e023324a2a8b02
I20240501 08:12:58.202225 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet 220add4033694ff381e023324a2a8b02 (table rebalancer_extra_table_38 [id=099418eaa97444bf9763310daefd6f13]) successfully deleted
I20240501 08:12:58.554461 30669 raft_consensus.cc:1203] T 52b69ecbe4094f32a1ed73b71b934451 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:58.686724 28020 raft_consensus.cc:1203] T bc84d1d413994a76bd6cf42335438755 P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:59.074092 3413 raft_consensus.cc:1050] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5: attempting to promote NON_VOTER aad81047009a4ffcbc450b60030583b5 to VOTER
I20240501 08:12:59.074910 3413 consensus_queue.cc:237] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } }
I20240501 08:12:59.083329 28022 raft_consensus.cc:1261] T bc84d1d413994a76bd6cf42335438755 P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:59.083624 27892 raft_consensus.cc:1261] T bc84d1d413994a76bd6cf42335438755 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:59.083678 30536 raft_consensus.cc:1261] T bc84d1d413994a76bd6cf42335438755 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:59.084136 3109 consensus_queue.cc:1035] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:59.085074 3509 consensus_queue.cc:1035] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:59.085566 3509 consensus_queue.cc:1035] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:59.089805 3509 raft_consensus.cc:2932] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:59.093549 30536 raft_consensus.cc:2932] T bc84d1d413994a76bd6cf42335438755 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:59.097342 27892 raft_consensus.cc:2932] T bc84d1d413994a76bd6cf42335438755 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:59.097347 28020 raft_consensus.cc:2932] T bc84d1d413994a76bd6cf42335438755 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:59.098464 27618 catalog_manager.cc:5484] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index 2 to 3, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:12:59.109323 30802 consensus_queue.cc:237] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } }
I20240501 08:12:59.111157 28020 raft_consensus.cc:1261] T bc84d1d413994a76bd6cf42335438755 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:59.111580 30536 raft_consensus.cc:1261] T bc84d1d413994a76bd6cf42335438755 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:59.111681 3509 consensus_queue.cc:1035] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:59.112514 3509 consensus_queue.cc:1035] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:59.115553 3109 raft_consensus.cc:2932] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:59.119660 27605 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bc84d1d413994a76bd6cf42335438755 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:59.122949 27615 catalog_manager.cc:5484] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index 3 to 4, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:59.122969 30536 raft_consensus.cc:2932] T bc84d1d413994a76bd6cf42335438755 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:59.127667 28022 raft_consensus.cc:2932] T bc84d1d413994a76bd6cf42335438755 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:12:59.131217 27871 tablet_service.cc:1514] Processing DeleteTablet for tablet bc84d1d413994a76bd6cf42335438755 with delete_type TABLET_DATA_TOMBSTONED (TS f1525cbc82a24d13b65f136695c903af not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:59.134975 3543 raft_consensus.cc:1050] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5: attempting to promote NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 to VOTER
I20240501 08:12:59.140172 3541 tablet_replica.cc:330] T bc84d1d413994a76bd6cf42335438755 P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:59.140398 3541 raft_consensus.cc:2229] T bc84d1d413994a76bd6cf42335438755 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:59.140617 3541 raft_consensus.cc:2258] T bc84d1d413994a76bd6cf42335438755 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:59.141577 3541 ts_tablet_manager.cc:1875] T bc84d1d413994a76bd6cf42335438755 P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:59.142345 3543 consensus_queue.cc:237] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:12:59.144814 30672 raft_consensus.cc:1261] T 52b69ecbe4094f32a1ed73b71b934451 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:59.144932 28152 raft_consensus.cc:1261] T 52b69ecbe4094f32a1ed73b71b934451 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:59.145318 27892 raft_consensus.cc:1261] T 52b69ecbe4094f32a1ed73b71b934451 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:59.145673 3542 consensus_queue.cc:1035] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:59.146051 3508 consensus_queue.cc:1035] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:59.146281 3543 consensus_queue.cc:1035] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:59.156338 3542 raft_consensus.cc:2932] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:59.158821 3541 ts_tablet_manager.cc:1888] T bc84d1d413994a76bd6cf42335438755 P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:12:59.160948 28152 raft_consensus.cc:2932] T 52b69ecbe4094f32a1ed73b71b934451 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:59.161468 27892 raft_consensus.cc:2932] T 52b69ecbe4094f32a1ed73b71b934451 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:59.166625 3541 log.cc:1196] T bc84d1d413994a76bd6cf42335438755 P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/bc84d1d413994a76bd6cf42335438755
I20240501 08:12:59.166461 27615 catalog_manager.cc:5484] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 4 to 5, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:12:59.168476 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet bc84d1d413994a76bd6cf42335438755 (table rebalancer_extra_table_2 [id=6fb79b6584164a1c9205937b7afef405]) successfully deleted
I20240501 08:12:59.177982 30669 raft_consensus.cc:2932] T 52b69ecbe4094f32a1ed73b71b934451 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:59.191896 28022 consensus_queue.cc:237] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:12:59.194870 30669 raft_consensus.cc:1261] T 52b69ecbe4094f32a1ed73b71b934451 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:59.195468 28152 raft_consensus.cc:1261] T 52b69ecbe4094f32a1ed73b71b934451 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:12:59.195432 3508 consensus_queue.cc:1035] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:59.196015 3540 consensus_queue.cc:1035] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:12:59.199856 3540 raft_consensus.cc:2932] T 52b69ecbe4094f32a1ed73b71b934451 P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:59.200809 30669 raft_consensus.cc:2932] T 52b69ecbe4094f32a1ed73b71b934451 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:59.203955 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 52b69ecbe4094f32a1ed73b71b934451 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:12:59.206528 28152 raft_consensus.cc:2932] T 52b69ecbe4094f32a1ed73b71b934451 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:59.212033 27615 catalog_manager.cc:5484] T 52b69ecbe4094f32a1ed73b71b934451 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 5 to 6, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:12:59.217567 27871 tablet_service.cc:1514] Processing DeleteTablet for tablet 52b69ecbe4094f32a1ed73b71b934451 with delete_type TABLET_DATA_TOMBSTONED (TS f1525cbc82a24d13b65f136695c903af not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48054
I20240501 08:12:59.218003 3541 tablet_replica.cc:330] T 52b69ecbe4094f32a1ed73b71b934451 P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:12:59.218281 3541 raft_consensus.cc:2229] T 52b69ecbe4094f32a1ed73b71b934451 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:59.218535 3541 raft_consensus.cc:2258] T 52b69ecbe4094f32a1ed73b71b934451 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:59.219820 3541 ts_tablet_manager.cc:1875] T 52b69ecbe4094f32a1ed73b71b934451 P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:59.226325 3541 ts_tablet_manager.cc:1888] T 52b69ecbe4094f32a1ed73b71b934451 P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 08:12:59.226534 3541 log.cc:1196] T 52b69ecbe4094f32a1ed73b71b934451 P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/52b69ecbe4094f32a1ed73b71b934451
I20240501 08:12:59.227391 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet 52b69ecbe4094f32a1ed73b71b934451 (table rebalance_test_table_11 [id=05c446d8d19b4077b3441b0349357c6e]) successfully deleted
I20240501 08:12:59.749702 30514 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:12:59.758921 27741 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:12:59.808022 30784 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 08:12:59.839737 28131 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 08:12:59.881269 30650 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:59.904081 27871 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 08:12:59.946017 30916 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:12:59.961902 28000 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:00.126034 30802 consensus_queue.cc:237] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:13:00.128440 30934 raft_consensus.cc:1261] T f607d2e086cc4352a05a5ef2b02788cc P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 08:13:00.128545 30739 consensus_peers.cc:485] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f607d2e086cc4352a05a5ef2b02788cc. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:00.128921 3632 consensus_queue.cc:1035] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:13:00.129406 27891 raft_consensus.cc:1261] T f607d2e086cc4352a05a5ef2b02788cc P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:13:00.129962 3533 consensus_queue.cc:1035] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:13:00.144815 3109 raft_consensus.cc:2932] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:13:00.145711 30936 raft_consensus.cc:2932] T f607d2e086cc4352a05a5ef2b02788cc P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:13:00.151949 27619 catalog_manager.cc:5484] T f607d2e086cc4352a05a5ef2b02788cc P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index 4 to 5, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:13:00.152371 27892 raft_consensus.cc:2932] T f607d2e086cc4352a05a5ef2b02788cc P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:13:00.177806 27760 consensus_queue.cc:237] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
I20240501 08:13:00.181003 30669 raft_consensus.cc:1261] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:13:00.181463 27892 raft_consensus.cc:1261] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:13:00.181849 3246 consensus_queue.cc:1035] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 08:13:00.182301 27697 consensus_peers.cc:485] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 -> Peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): Couldn't send request to peer 31811708c6774037b82df44024b229b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 802538381de843ab8d326014a0b4ec15. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:00.182467 2933 consensus_queue.cc:1035] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:13:00.186694 3246 raft_consensus.cc:2932] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:13:00.192585 30669 raft_consensus.cc:2932] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:13:00.193595 27892 raft_consensus.cc:2932] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:13:00.202160 27615 catalog_manager.cc:5484] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af reported cstate change: config changed from index 6 to 7, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:13:00.230000 28152 consensus_queue.cc:237] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
I20240501 08:13:00.233026 30934 raft_consensus.cc:1261] T 4b3e3d39f82549079293d98357884bf0 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 08:13:00.233089 28086 consensus_peers.cc:485] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 -> Peer 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): Couldn't send request to peer 64998323a1f4482d8704bf20594d13a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b3e3d39f82549079293d98357884bf0. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:00.233696 3475 consensus_queue.cc:1035] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:13:00.233919 28020 raft_consensus.cc:1261] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:13:00.234788 3475 consensus_queue.cc:1035] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:13:00.238719 3354 raft_consensus.cc:2932] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:13:00.242810 28020 raft_consensus.cc:2932] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:13:00.251497 27615 catalog_manager.cc:5484] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 6 to 7, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:13:00.259182 30934 raft_consensus.cc:2932] T 4b3e3d39f82549079293d98357884bf0 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:13:00.284801 30802 consensus_queue.cc:237] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:13:00.287005 30669 raft_consensus.cc:1261] T 00205a461b704a19bbd39bc71361e529 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:13:00.287005 27760 raft_consensus.cc:1261] T 00205a461b704a19bbd39bc71361e529 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 08:13:00.287181 30737 consensus_peers.cc:485] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 -> Peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Couldn't send request to peer 91d11eae5ef6432ca86eb8ed446b6ea6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00205a461b704a19bbd39bc71361e529. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:00.287549 3533 consensus_queue.cc:1035] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:13:00.287835 3632 consensus_queue.cc:1035] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:13:00.290470 3632 raft_consensus.cc:2932] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:13:00.291504 30669 raft_consensus.cc:2932] T 00205a461b704a19bbd39bc71361e529 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:13:00.291726 27760 raft_consensus.cc:2932] T 00205a461b704a19bbd39bc71361e529 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:13:00.298642 27616 catalog_manager.cc:5484] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index 4 to 5, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:00.330729 27892 consensus_queue.cc:237] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:13:00.336118 27760 raft_consensus.cc:1261] T 7f7211c8c0e041bf89930aa50a4438ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:13:00.337646 30536 raft_consensus.cc:1261] T 7f7211c8c0e041bf89930aa50a4438ea P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 08:13:00.338101 27827 consensus_peers.cc:485] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af -> Peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): Couldn't send request to peer a29cd590999b4f32849ff5689cd53c11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f7211c8c0e041bf89930aa50a4438ea. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:00.338433 2620 consensus_queue.cc:1035] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:13:00.340973 2620 consensus_queue.cc:1035] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:13:00.346635 2620 raft_consensus.cc:2932] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:13:00.352516 27760 raft_consensus.cc:2932] T 7f7211c8c0e041bf89930aa50a4438ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:13:00.357355 27619 catalog_manager.cc:5484] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af reported cstate change: config changed from index 6 to 7, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:00.373116 30536 raft_consensus.cc:2932] T 7f7211c8c0e041bf89930aa50a4438ea P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:13:00.411366 28020 consensus_queue.cc:237] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:13:00.414412 27892 raft_consensus.cc:1261] T b12559fc416a4f3cbffa018611226bcc P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:13:00.414600 30934 raft_consensus.cc:1261] T b12559fc416a4f3cbffa018611226bcc P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:13:00.414956 3543 consensus_queue.cc:1035] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:13:00.415414 3542 consensus_queue.cc:1035] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 08:13:00.416472 27955 consensus_peers.cc:485] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 -> Peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Couldn't send request to peer 9a022e1a501f4d44a0468054d6c3de47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b12559fc416a4f3cbffa018611226bcc. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:00.423702 3542 raft_consensus.cc:2932] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:13:00.424533 30934 raft_consensus.cc:2932] T b12559fc416a4f3cbffa018611226bcc P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:13:00.424943 27892 raft_consensus.cc:2932] T b12559fc416a4f3cbffa018611226bcc P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:13:00.437112 27615 catalog_manager.cc:5484] T b12559fc416a4f3cbffa018611226bcc P f1525cbc82a24d13b65f136695c903af reported cstate change: config changed from index 6 to 7, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:13:00.486829 28020 consensus_queue.cc:237] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
I20240501 08:13:00.489742 30669 raft_consensus.cc:1261] T e6fb30f10c754d3a8e73e124ec59768b P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 08:13:00.489782 27958 consensus_peers.cc:485] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 -> Peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): Couldn't send request to peer 31811708c6774037b82df44024b229b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6fb30f10c754d3a8e73e124ec59768b. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:00.490152 28151 raft_consensus.cc:1261] T e6fb30f10c754d3a8e73e124ec59768b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:13:00.490291 3543 consensus_queue.cc:1035] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:13:00.490657 3688 consensus_queue.cc:1035] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:13:00.493520 3688 raft_consensus.cc:2932] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:13:00.495330 30669 raft_consensus.cc:2932] T e6fb30f10c754d3a8e73e124ec59768b P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:13:00.508823 27616 catalog_manager.cc:5484] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 6 to 7, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:00.514178 28151 raft_consensus.cc:2932] T e6fb30f10c754d3a8e73e124ec59768b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:13:00.523507 3704 ts_tablet_manager.cc:902] T f607d2e086cc4352a05a5ef2b02788cc P 6331f77dae9d452184e8cd0cbf5f1b43: Initiating tablet copy from peer 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:13:00.525547 3704 tablet_copy_client.cc:308] T f607d2e086cc4352a05a5ef2b02788cc P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.135:33503
I20240501 08:13:00.526358 30825 tablet_copy_service.cc:140] P 64998323a1f4482d8704bf20594d13a5: Received BeginTabletCopySession request for tablet f607d2e086cc4352a05a5ef2b02788cc from peer 6331f77dae9d452184e8cd0cbf5f1b43 ({username='slave'} at 127.21.84.129:58333)
I20240501 08:13:00.526511 30825 tablet_copy_service.cc:161] P 64998323a1f4482d8704bf20594d13a5: Beginning new tablet copy session on tablet f607d2e086cc4352a05a5ef2b02788cc from peer 6331f77dae9d452184e8cd0cbf5f1b43 at {username='slave'} at 127.21.84.129:58333: session id = 6331f77dae9d452184e8cd0cbf5f1b43-f607d2e086cc4352a05a5ef2b02788cc
I20240501 08:13:00.529354 30825 tablet_copy_source_session.cc:205] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:00.531510 3704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f607d2e086cc4352a05a5ef2b02788cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:00.532363 27892 consensus_queue.cc:237] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
W20240501 08:13:00.535250 27826 consensus_peers.cc:485] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af -> Peer 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): Couldn't send request to peer 64998323a1f4482d8704bf20594d13a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9820951bffc749139229e2b79c8ab597. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:00.535476 30669 raft_consensus.cc:1261] T 9820951bffc749139229e2b79c8ab597 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:13:00.535957 28020 raft_consensus.cc:1261] T 9820951bffc749139229e2b79c8ab597 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:13:00.535931 3531 consensus_queue.cc:1035] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:13:00.536446 3531 consensus_queue.cc:1035] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:13:00.537859 3704 tablet_copy_client.cc:783] T f607d2e086cc4352a05a5ef2b02788cc P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:00.538136 3704 tablet_copy_client.cc:647] T f607d2e086cc4352a05a5ef2b02788cc P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:00.540376 3704 tablet_copy_client.cc:523] T f607d2e086cc4352a05a5ef2b02788cc P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:00.541435 3531 raft_consensus.cc:2932] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:13:00.544560 3704 tablet_bootstrap.cc:492] T f607d2e086cc4352a05a5ef2b02788cc P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:13:00.546646 28020 raft_consensus.cc:2932] T 9820951bffc749139229e2b79c8ab597 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:13:00.549739 30669 raft_consensus.cc:2932] T 9820951bffc749139229e2b79c8ab597 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:13:00.554005 27615 catalog_manager.cc:5484] T 9820951bffc749139229e2b79c8ab597 P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 6 to 7, NON_VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) added. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } } }
I20240501 08:13:00.602119 3704 tablet_bootstrap.cc:492] T f607d2e086cc4352a05a5ef2b02788cc P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:00.603547 3704 tablet_bootstrap.cc:492] T f607d2e086cc4352a05a5ef2b02788cc P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap complete.
I20240501 08:13:00.603878 3704 ts_tablet_manager.cc:1367] T f607d2e086cc4352a05a5ef2b02788cc P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.059s user 0.019s sys 0.030s
I20240501 08:13:00.604661 3704 raft_consensus.cc:348] T f607d2e086cc4352a05a5ef2b02788cc P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:13:00.605041 3704 raft_consensus.cc:729] T f607d2e086cc4352a05a5ef2b02788cc P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: LEARNER
I20240501 08:13:00.605244 3704 consensus_queue.cc:260] T f607d2e086cc4352a05a5ef2b02788cc P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:13:00.606595 3704 ts_tablet_manager.cc:1398] T f607d2e086cc4352a05a5ef2b02788cc P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:13:00.607220 30825 tablet_copy_service.cc:342] P 64998323a1f4482d8704bf20594d13a5: Request end of tablet copy session 6331f77dae9d452184e8cd0cbf5f1b43-f607d2e086cc4352a05a5ef2b02788cc received from {username='slave'} at 127.21.84.129:58333
I20240501 08:13:00.607333 30825 tablet_copy_service.cc:433] P 64998323a1f4482d8704bf20594d13a5: ending tablet copy session 6331f77dae9d452184e8cd0cbf5f1b43-f607d2e086cc4352a05a5ef2b02788cc on tablet f607d2e086cc4352a05a5ef2b02788cc with peer 6331f77dae9d452184e8cd0cbf5f1b43
I20240501 08:13:00.628811 28020 consensus_queue.cc:237] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:13:00.631397 30537 raft_consensus.cc:1261] T 6c04880ea87d4894bd3a63d94f4a69f4 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 08:13:00.631584 27955 consensus_peers.cc:485] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 -> Peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Couldn't send request to peer 9a022e1a501f4d44a0468054d6c3de47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c04880ea87d4894bd3a63d94f4a69f4. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:00.631695 27758 raft_consensus.cc:1261] T 6c04880ea87d4894bd3a63d94f4a69f4 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:13:00.631951 3688 consensus_queue.cc:1035] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:13:00.632263 3543 consensus_queue.cc:1035] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:13:00.634909 3543 raft_consensus.cc:2932] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:13:00.635473 27758 raft_consensus.cc:2932] T 6c04880ea87d4894bd3a63d94f4a69f4 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:13:00.639024 30537 raft_consensus.cc:2932] T 6c04880ea87d4894bd3a63d94f4a69f4 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:13:00.643009 27619 catalog_manager.cc:5484] T 6c04880ea87d4894bd3a63d94f4a69f4 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index 6 to 7, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:13:00.677304 3730 ts_tablet_manager.cc:902] T 4b3e3d39f82549079293d98357884bf0 P 64998323a1f4482d8704bf20594d13a5: Initiating tablet copy from peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:13:00.679507 3730 tablet_copy_client.cc:308] T 4b3e3d39f82549079293d98357884bf0 P 64998323a1f4482d8704bf20594d13a5: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.132:41323
I20240501 08:13:00.680194 28171 tablet_copy_service.cc:140] P 91d11eae5ef6432ca86eb8ed446b6ea6: Received BeginTabletCopySession request for tablet 4b3e3d39f82549079293d98357884bf0 from peer 64998323a1f4482d8704bf20594d13a5 ({username='slave'} at 127.21.84.135:48917)
I20240501 08:13:00.680320 28171 tablet_copy_service.cc:161] P 91d11eae5ef6432ca86eb8ed446b6ea6: Beginning new tablet copy session on tablet 4b3e3d39f82549079293d98357884bf0 from peer 64998323a1f4482d8704bf20594d13a5 at {username='slave'} at 127.21.84.135:48917: session id = 64998323a1f4482d8704bf20594d13a5-4b3e3d39f82549079293d98357884bf0
I20240501 08:13:00.682216 28171 tablet_copy_source_session.cc:205] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:00.683939 3730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b3e3d39f82549079293d98357884bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:00.684754 30934 consensus_queue.cc:237] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:13:00.689778 3730 tablet_copy_client.cc:783] T 4b3e3d39f82549079293d98357884bf0 P 64998323a1f4482d8704bf20594d13a5: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:00.690259 3730 tablet_copy_client.cc:647] T 4b3e3d39f82549079293d98357884bf0 P 64998323a1f4482d8704bf20594d13a5: tablet copy: Starting download of 1 WAL segments...
W20240501 08:13:00.691731 30870 consensus_peers.cc:485] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 -> Peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Couldn't send request to peer 91d11eae5ef6432ca86eb8ed446b6ea6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f63215605ade41a5bc8d97a35ce33a31. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:00.691764 27758 raft_consensus.cc:1261] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:13:00.692091 30537 raft_consensus.cc:1261] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:13:00.692620 3722 consensus_queue.cc:1035] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:13:00.692870 3236 consensus_queue.cc:1035] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:13:00.693161 3730 tablet_copy_client.cc:523] T 4b3e3d39f82549079293d98357884bf0 P 64998323a1f4482d8704bf20594d13a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:00.696502 3730 tablet_bootstrap.cc:492] T 4b3e3d39f82549079293d98357884bf0 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:13:00.697386 3567 raft_consensus.cc:2932] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:13:00.704609 30537 raft_consensus.cc:2932] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:13:00.705025 27758 raft_consensus.cc:2932] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:13:00.706157 27619 catalog_manager.cc:5484] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index 4 to 5, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:00.749477 30934 consensus_queue.cc:237] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:13:00.765486 3745 ts_tablet_manager.cc:902] T 802538381de843ab8d326014a0b4ec15 P 31811708c6774037b82df44024b229b6: Initiating tablet copy from peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
W20240501 08:13:00.766431 30872 consensus_peers.cc:485] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6b0bcb4d07e4125bccf1711a094576e. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:00.767587 30669 raft_consensus.cc:1261] T b6b0bcb4d07e4125bccf1711a094576e P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:00.768685 3236 consensus_queue.cc:1035] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:00.769075 30536 raft_consensus.cc:1261] T b6b0bcb4d07e4125bccf1711a094576e P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:00.770511 3236 consensus_queue.cc:1035] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:00.772202 3236 raft_consensus.cc:2932] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:13:00.780514 30669 raft_consensus.cc:2932] T b6b0bcb4d07e4125bccf1711a094576e P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:13:00.788309 30536 raft_consensus.cc:2932] T b6b0bcb4d07e4125bccf1711a094576e P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } } }
I20240501 08:13:00.793722 3730 tablet_bootstrap.cc:492] T 4b3e3d39f82549079293d98357884bf0 P 64998323a1f4482d8704bf20594d13a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:00.794152 3730 tablet_bootstrap.cc:492] T 4b3e3d39f82549079293d98357884bf0 P 64998323a1f4482d8704bf20594d13a5: Bootstrap complete.
I20240501 08:13:00.794158 3745 tablet_copy_client.cc:308] T 802538381de843ab8d326014a0b4ec15 P 31811708c6774037b82df44024b229b6: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.129:43471
I20240501 08:13:00.794405 3730 ts_tablet_manager.cc:1367] T 4b3e3d39f82549079293d98357884bf0 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.098s user 0.065s sys 0.006s
I20240501 08:13:00.795049 3730 raft_consensus.cc:348] T 4b3e3d39f82549079293d98357884bf0 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
I20240501 08:13:00.794926 27616 catalog_manager.cc:5484] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index -1 to 2, NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) added. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:00.795228 3730 raft_consensus.cc:729] T 4b3e3d39f82549079293d98357884bf0 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: LEARNER
I20240501 08:13:00.795405 3730 consensus_queue.cc:260] T 4b3e3d39f82549079293d98357884bf0 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
I20240501 08:13:00.795960 3730 ts_tablet_manager.cc:1398] T 4b3e3d39f82549079293d98357884bf0 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:13:00.796157 27781 tablet_copy_service.cc:140] P 6331f77dae9d452184e8cd0cbf5f1b43: Received BeginTabletCopySession request for tablet 802538381de843ab8d326014a0b4ec15 from peer 31811708c6774037b82df44024b229b6 ({username='slave'} at 127.21.84.133:35347)
I20240501 08:13:00.796308 27781 tablet_copy_service.cc:161] P 6331f77dae9d452184e8cd0cbf5f1b43: Beginning new tablet copy session on tablet 802538381de843ab8d326014a0b4ec15 from peer 31811708c6774037b82df44024b229b6 at {username='slave'} at 127.21.84.133:35347: session id = 31811708c6774037b82df44024b229b6-802538381de843ab8d326014a0b4ec15
I20240501 08:13:00.796658 28171 tablet_copy_service.cc:342] P 91d11eae5ef6432ca86eb8ed446b6ea6: Request end of tablet copy session 64998323a1f4482d8704bf20594d13a5-4b3e3d39f82549079293d98357884bf0 received from {username='slave'} at 127.21.84.135:48917
I20240501 08:13:00.796756 28171 tablet_copy_service.cc:433] P 91d11eae5ef6432ca86eb8ed446b6ea6: ending tablet copy session 64998323a1f4482d8704bf20594d13a5-4b3e3d39f82549079293d98357884bf0 on tablet 4b3e3d39f82549079293d98357884bf0 with peer 64998323a1f4482d8704bf20594d13a5
I20240501 08:13:00.800134 27781 tablet_copy_source_session.cc:205] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:00.809024 3745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 802538381de843ab8d326014a0b4ec15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:00.820437 3757 ts_tablet_manager.cc:902] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11: Initiating tablet copy from peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:13:00.823508 3745 tablet_copy_client.cc:783] T 802538381de843ab8d326014a0b4ec15 P 31811708c6774037b82df44024b229b6: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:00.823880 3745 tablet_copy_client.cc:647] T 802538381de843ab8d326014a0b4ec15 P 31811708c6774037b82df44024b229b6: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:00.837591 3764 ts_tablet_manager.cc:902] T 00205a461b704a19bbd39bc71361e529 P 91d11eae5ef6432ca86eb8ed446b6ea6: Initiating tablet copy from peer 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:13:00.844846 3757 tablet_copy_client.cc:308] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.130:36503
I20240501 08:13:00.845654 27912 tablet_copy_service.cc:140] P f1525cbc82a24d13b65f136695c903af: Received BeginTabletCopySession request for tablet 7f7211c8c0e041bf89930aa50a4438ea from peer a29cd590999b4f32849ff5689cd53c11 ({username='slave'} at 127.21.84.136:54747)
I20240501 08:13:00.845839 27912 tablet_copy_service.cc:161] P f1525cbc82a24d13b65f136695c903af: Beginning new tablet copy session on tablet 7f7211c8c0e041bf89930aa50a4438ea from peer a29cd590999b4f32849ff5689cd53c11 at {username='slave'} at 127.21.84.136:54747: session id = a29cd590999b4f32849ff5689cd53c11-7f7211c8c0e041bf89930aa50a4438ea
I20240501 08:13:00.856909 27912 tablet_copy_source_session.cc:205] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:00.859622 3757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f7211c8c0e041bf89930aa50a4438ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:00.861030 3745 tablet_copy_client.cc:523] T 802538381de843ab8d326014a0b4ec15 P 31811708c6774037b82df44024b229b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:00.864693 3745 tablet_bootstrap.cc:492] T 802538381de843ab8d326014a0b4ec15 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:13:00.866078 3757 tablet_copy_client.cc:783] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:00.866298 3757 tablet_copy_client.cc:647] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:00.866755 28152 consensus_queue.cc:237] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:13:00.869513 3757 tablet_copy_client.cc:523] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:00.873067 3757 tablet_bootstrap.cc:492] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:13:00.881050 3764 tablet_copy_client.cc:308] T 00205a461b704a19bbd39bc71361e529 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.135:33503
W20240501 08:13:00.882370 28085 consensus_peers.cc:485] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 -> Peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Couldn't send request to peer 9a022e1a501f4d44a0468054d6c3de47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bc5778711a4c479881088af790f5b7d5. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:00.883042 30825 tablet_copy_service.cc:140] P 64998323a1f4482d8704bf20594d13a5: Received BeginTabletCopySession request for tablet 00205a461b704a19bbd39bc71361e529 from peer 91d11eae5ef6432ca86eb8ed446b6ea6 ({username='slave'} at 127.21.84.132:54591)
I20240501 08:13:00.883204 30825 tablet_copy_service.cc:161] P 64998323a1f4482d8704bf20594d13a5: Beginning new tablet copy session on tablet 00205a461b704a19bbd39bc71361e529 from peer 91d11eae5ef6432ca86eb8ed446b6ea6 at {username='slave'} at 127.21.84.132:54591: session id = 91d11eae5ef6432ca86eb8ed446b6ea6-00205a461b704a19bbd39bc71361e529
I20240501 08:13:00.888458 30536 raft_consensus.cc:1261] T bc5778711a4c479881088af790f5b7d5 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:13:00.890576 3354 consensus_queue.cc:1035] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:13:00.893693 3773 ts_tablet_manager.cc:902] T b12559fc416a4f3cbffa018611226bcc P 9a022e1a501f4d44a0468054d6c3de47: Initiating tablet copy from peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:13:00.894707 30825 tablet_copy_source_session.cc:205] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:00.896759 3764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00205a461b704a19bbd39bc71361e529. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:00.900972 28022 raft_consensus.cc:1261] T bc5778711a4c479881088af790f5b7d5 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:13:00.902014 3664 consensus_queue.cc:1035] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:13:00.903360 3764 tablet_copy_client.cc:783] T 00205a461b704a19bbd39bc71361e529 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:00.903602 3764 tablet_copy_client.cc:647] T 00205a461b704a19bbd39bc71361e529 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:00.906179 3764 tablet_copy_client.cc:523] T 00205a461b704a19bbd39bc71361e529 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:00.908706 3664 raft_consensus.cc:2932] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:13:00.909536 3764 tablet_bootstrap.cc:492] T 00205a461b704a19bbd39bc71361e529 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:13:00.911159 30536 raft_consensus.cc:2932] T bc5778711a4c479881088af790f5b7d5 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:13:00.911870 28020 raft_consensus.cc:2932] T bc5778711a4c479881088af790f5b7d5 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:13:00.918960 27619 catalog_manager.cc:5484] T bc5778711a4c479881088af790f5b7d5 P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 6 to 7, NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) added. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } } }
I20240501 08:13:00.942318 3764 tablet_bootstrap.cc:492] T 00205a461b704a19bbd39bc71361e529 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:00.942740 3764 tablet_bootstrap.cc:492] T 00205a461b704a19bbd39bc71361e529 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap complete.
I20240501 08:13:00.943032 3764 ts_tablet_manager.cc:1367] T 00205a461b704a19bbd39bc71361e529 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.034s user 0.014s sys 0.002s
I20240501 08:13:00.943926 3764 raft_consensus.cc:348] T 00205a461b704a19bbd39bc71361e529 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:13:00.944159 3764 raft_consensus.cc:729] T 00205a461b704a19bbd39bc71361e529 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: LEARNER
I20240501 08:13:00.944407 3764 consensus_queue.cc:260] T 00205a461b704a19bbd39bc71361e529 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:13:00.957211 3773 tablet_copy_client.cc:308] T b12559fc416a4f3cbffa018611226bcc P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.131:36305
I20240501 08:13:00.958421 28042 tablet_copy_service.cc:140] P aad81047009a4ffcbc450b60030583b5: Received BeginTabletCopySession request for tablet b12559fc416a4f3cbffa018611226bcc from peer 9a022e1a501f4d44a0468054d6c3de47 ({username='slave'} at 127.21.84.134:40301)
I20240501 08:13:00.958578 28042 tablet_copy_service.cc:161] P aad81047009a4ffcbc450b60030583b5: Beginning new tablet copy session on tablet b12559fc416a4f3cbffa018611226bcc from peer 9a022e1a501f4d44a0468054d6c3de47 at {username='slave'} at 127.21.84.134:40301: session id = 9a022e1a501f4d44a0468054d6c3de47-b12559fc416a4f3cbffa018611226bcc
I20240501 08:13:00.958746 3764 ts_tablet_manager.cc:1398] T 00205a461b704a19bbd39bc71361e529 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20240501 08:13:00.959503 30825 tablet_copy_service.cc:342] P 64998323a1f4482d8704bf20594d13a5: Request end of tablet copy session 91d11eae5ef6432ca86eb8ed446b6ea6-00205a461b704a19bbd39bc71361e529 received from {username='slave'} at 127.21.84.132:54591
I20240501 08:13:00.959621 30825 tablet_copy_service.cc:433] P 64998323a1f4482d8704bf20594d13a5: ending tablet copy session 91d11eae5ef6432ca86eb8ed446b6ea6-00205a461b704a19bbd39bc71361e529 on tablet 00205a461b704a19bbd39bc71361e529 with peer 91d11eae5ef6432ca86eb8ed446b6ea6
I20240501 08:13:00.961638 28042 tablet_copy_source_session.cc:205] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:00.964071 3757 tablet_bootstrap.cc:492] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:00.964529 3757 tablet_bootstrap.cc:492] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11: Bootstrap complete.
I20240501 08:13:00.964800 3757 ts_tablet_manager.cc:1367] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.092s user 0.056s sys 0.001s
I20240501 08:13:00.965595 3757 raft_consensus.cc:348] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:13:00.965842 3757 raft_consensus.cc:729] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: LEARNER
I20240501 08:13:00.966081 3757 consensus_queue.cc:260] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:13:00.966797 3757 ts_tablet_manager.cc:1398] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240501 08:13:00.967075 3773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b12559fc416a4f3cbffa018611226bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:00.967514 27912 tablet_copy_service.cc:342] P f1525cbc82a24d13b65f136695c903af: Request end of tablet copy session a29cd590999b4f32849ff5689cd53c11-7f7211c8c0e041bf89930aa50a4438ea received from {username='slave'} at 127.21.84.136:54747
I20240501 08:13:00.967613 27912 tablet_copy_service.cc:433] P f1525cbc82a24d13b65f136695c903af: ending tablet copy session a29cd590999b4f32849ff5689cd53c11-7f7211c8c0e041bf89930aa50a4438ea on tablet 7f7211c8c0e041bf89930aa50a4438ea with peer a29cd590999b4f32849ff5689cd53c11
I20240501 08:13:00.982265 3773 tablet_copy_client.cc:783] T b12559fc416a4f3cbffa018611226bcc P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:00.982535 3773 tablet_copy_client.cc:647] T b12559fc416a4f3cbffa018611226bcc P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:00.988577 3773 tablet_copy_client.cc:523] T b12559fc416a4f3cbffa018611226bcc P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:00.992988 3773 tablet_bootstrap.cc:492] T b12559fc416a4f3cbffa018611226bcc P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:13:01.015408 27889 consensus_queue.cc:237] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:13:01.018810 28020 raft_consensus.cc:1261] T 0e06cf862c954e8fb1ca087ec5184497 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:13:01.019507 2620 consensus_queue.cc:1035] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:13:01.019634 3787 ts_tablet_manager.cc:902] T e6fb30f10c754d3a8e73e124ec59768b P 31811708c6774037b82df44024b229b6: Initiating tablet copy from peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
W20240501 08:13:01.023239 27827 consensus_peers.cc:485] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af -> Peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): Couldn't send request to peer a29cd590999b4f32849ff5689cd53c11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e06cf862c954e8fb1ca087ec5184497. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:01.023425 30802 raft_consensus.cc:1261] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:13:01.024830 3531 consensus_queue.cc:1035] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:13:01.030369 3531 raft_consensus.cc:2932] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:13:01.032599 28020 raft_consensus.cc:2932] T 0e06cf862c954e8fb1ca087ec5184497 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:13:01.041620 30802 raft_consensus.cc:2932] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:13:01.047519 3787 tablet_copy_client.cc:308] T e6fb30f10c754d3a8e73e124ec59768b P 31811708c6774037b82df44024b229b6: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.131:36305
I20240501 08:13:01.049170 28042 tablet_copy_service.cc:140] P aad81047009a4ffcbc450b60030583b5: Received BeginTabletCopySession request for tablet e6fb30f10c754d3a8e73e124ec59768b from peer 31811708c6774037b82df44024b229b6 ({username='slave'} at 127.21.84.133:53413)
I20240501 08:13:01.049336 28042 tablet_copy_service.cc:161] P aad81047009a4ffcbc450b60030583b5: Beginning new tablet copy session on tablet e6fb30f10c754d3a8e73e124ec59768b from peer 31811708c6774037b82df44024b229b6 at {username='slave'} at 127.21.84.133:53413: session id = 31811708c6774037b82df44024b229b6-e6fb30f10c754d3a8e73e124ec59768b
I20240501 08:13:01.051074 27616 catalog_manager.cc:5484] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index 4 to 5, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:13:01.052145 28042 tablet_copy_source_session.cc:205] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:01.053478 3787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6fb30f10c754d3a8e73e124ec59768b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:01.061910 3787 tablet_copy_client.cc:783] T e6fb30f10c754d3a8e73e124ec59768b P 31811708c6774037b82df44024b229b6: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:01.062604 3787 tablet_copy_client.cc:647] T e6fb30f10c754d3a8e73e124ec59768b P 31811708c6774037b82df44024b229b6: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:01.064110 3787 tablet_copy_client.cc:523] T e6fb30f10c754d3a8e73e124ec59768b P 31811708c6774037b82df44024b229b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:01.071161 3787 tablet_bootstrap.cc:492] T e6fb30f10c754d3a8e73e124ec59768b P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:13:01.078668 3795 ts_tablet_manager.cc:902] T 6c04880ea87d4894bd3a63d94f4a69f4 P 9a022e1a501f4d44a0468054d6c3de47: Initiating tablet copy from peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:13:01.081835 3745 tablet_bootstrap.cc:492] T 802538381de843ab8d326014a0b4ec15 P 31811708c6774037b82df44024b229b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:01.082279 3745 tablet_bootstrap.cc:492] T 802538381de843ab8d326014a0b4ec15 P 31811708c6774037b82df44024b229b6: Bootstrap complete.
I20240501 08:13:01.082543 3745 ts_tablet_manager.cc:1367] T 802538381de843ab8d326014a0b4ec15 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.218s user 0.054s sys 0.010s
I20240501 08:13:01.083254 3745 raft_consensus.cc:348] T 802538381de843ab8d326014a0b4ec15 P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
I20240501 08:13:01.083499 3745 raft_consensus.cc:729] T 802538381de843ab8d326014a0b4ec15 P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: LEARNER
I20240501 08:13:01.089367 3773 tablet_bootstrap.cc:492] T b12559fc416a4f3cbffa018611226bcc P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:01.089848 3773 tablet_bootstrap.cc:492] T b12559fc416a4f3cbffa018611226bcc P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap complete.
I20240501 08:13:01.090214 3773 ts_tablet_manager.cc:1367] T b12559fc416a4f3cbffa018611226bcc P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.097s user 0.069s sys 0.007s
I20240501 08:13:01.091024 3773 raft_consensus.cc:348] T b12559fc416a4f3cbffa018611226bcc P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:13:01.092124 3773 raft_consensus.cc:729] T b12559fc416a4f3cbffa018611226bcc P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: LEARNER
I20240501 08:13:01.092387 3773 consensus_queue.cc:260] T b12559fc416a4f3cbffa018611226bcc P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:13:01.092494 3745 consensus_queue.cc:260] T 802538381de843ab8d326014a0b4ec15 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
I20240501 08:13:01.095345 3773 ts_tablet_manager.cc:1398] T b12559fc416a4f3cbffa018611226bcc P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 08:13:01.095995 28042 tablet_copy_service.cc:342] P aad81047009a4ffcbc450b60030583b5: Request end of tablet copy session 9a022e1a501f4d44a0468054d6c3de47-b12559fc416a4f3cbffa018611226bcc received from {username='slave'} at 127.21.84.134:40301
I20240501 08:13:01.096122 28042 tablet_copy_service.cc:433] P aad81047009a4ffcbc450b60030583b5: ending tablet copy session 9a022e1a501f4d44a0468054d6c3de47-b12559fc416a4f3cbffa018611226bcc on tablet b12559fc416a4f3cbffa018611226bcc with peer 9a022e1a501f4d44a0468054d6c3de47
I20240501 08:13:01.096143 3745 ts_tablet_manager.cc:1398] T 802538381de843ab8d326014a0b4ec15 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240501 08:13:01.097046 27781 tablet_copy_service.cc:342] P 6331f77dae9d452184e8cd0cbf5f1b43: Request end of tablet copy session 31811708c6774037b82df44024b229b6-802538381de843ab8d326014a0b4ec15 received from {username='slave'} at 127.21.84.133:35347
I20240501 08:13:01.097162 27781 tablet_copy_service.cc:433] P 6331f77dae9d452184e8cd0cbf5f1b43: ending tablet copy session 31811708c6774037b82df44024b229b6-802538381de843ab8d326014a0b4ec15 on tablet 802538381de843ab8d326014a0b4ec15 with peer 31811708c6774037b82df44024b229b6
I20240501 08:13:01.106963 27758 raft_consensus.cc:1203] T f607d2e086cc4352a05a5ef2b02788cc P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 08:13:01.137171 3795 tablet_copy_client.cc:308] T 6c04880ea87d4894bd3a63d94f4a69f4 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.131:36305
I20240501 08:13:01.139590 28042 tablet_copy_service.cc:140] P aad81047009a4ffcbc450b60030583b5: Received BeginTabletCopySession request for tablet 6c04880ea87d4894bd3a63d94f4a69f4 from peer 9a022e1a501f4d44a0468054d6c3de47 ({username='slave'} at 127.21.84.134:40301)
I20240501 08:13:01.139765 28042 tablet_copy_service.cc:161] P aad81047009a4ffcbc450b60030583b5: Beginning new tablet copy session on tablet 6c04880ea87d4894bd3a63d94f4a69f4 from peer 9a022e1a501f4d44a0468054d6c3de47 at {username='slave'} at 127.21.84.134:40301: session id = 9a022e1a501f4d44a0468054d6c3de47-6c04880ea87d4894bd3a63d94f4a69f4
I20240501 08:13:01.143499 28042 tablet_copy_source_session.cc:205] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:01.145148 3730 ts_tablet_manager.cc:902] T 9820951bffc749139229e2b79c8ab597 P 64998323a1f4482d8704bf20594d13a5: Initiating tablet copy from peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:13:01.146098 3795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c04880ea87d4894bd3a63d94f4a69f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:01.160027 3795 tablet_copy_client.cc:783] T 6c04880ea87d4894bd3a63d94f4a69f4 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:01.160297 3795 tablet_copy_client.cc:647] T 6c04880ea87d4894bd3a63d94f4a69f4 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:01.160769 3787 tablet_bootstrap.cc:492] T e6fb30f10c754d3a8e73e124ec59768b P 31811708c6774037b82df44024b229b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:01.162410 3795 tablet_copy_client.cc:523] T 6c04880ea87d4894bd3a63d94f4a69f4 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:01.162626 3787 tablet_bootstrap.cc:492] T e6fb30f10c754d3a8e73e124ec59768b P 31811708c6774037b82df44024b229b6: Bootstrap complete.
I20240501 08:13:01.162959 3787 ts_tablet_manager.cc:1367] T e6fb30f10c754d3a8e73e124ec59768b P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.092s user 0.051s sys 0.008s
I20240501 08:13:01.163997 3787 raft_consensus.cc:348] T e6fb30f10c754d3a8e73e124ec59768b P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
I20240501 08:13:01.164383 3787 raft_consensus.cc:729] T e6fb30f10c754d3a8e73e124ec59768b P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: LEARNER
I20240501 08:13:01.164615 3787 consensus_queue.cc:260] T e6fb30f10c754d3a8e73e124ec59768b P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
I20240501 08:13:01.168406 3787 ts_tablet_manager.cc:1398] T e6fb30f10c754d3a8e73e124ec59768b P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:13:01.170897 28042 tablet_copy_service.cc:342] P aad81047009a4ffcbc450b60030583b5: Request end of tablet copy session 31811708c6774037b82df44024b229b6-e6fb30f10c754d3a8e73e124ec59768b received from {username='slave'} at 127.21.84.133:53413
I20240501 08:13:01.171057 28042 tablet_copy_service.cc:433] P aad81047009a4ffcbc450b60030583b5: ending tablet copy session 31811708c6774037b82df44024b229b6-e6fb30f10c754d3a8e73e124ec59768b on tablet e6fb30f10c754d3a8e73e124ec59768b with peer 31811708c6774037b82df44024b229b6
I20240501 08:13:01.174783 30536 raft_consensus.cc:1203] T 802538381de843ab8d326014a0b4ec15 P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 08:13:01.181180 3795 tablet_bootstrap.cc:492] T 6c04880ea87d4894bd3a63d94f4a69f4 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:13:01.188845 3730 tablet_copy_client.cc:308] T 9820951bffc749139229e2b79c8ab597 P 64998323a1f4482d8704bf20594d13a5: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.130:36503
I20240501 08:13:01.190228 27912 tablet_copy_service.cc:140] P f1525cbc82a24d13b65f136695c903af: Received BeginTabletCopySession request for tablet 9820951bffc749139229e2b79c8ab597 from peer 64998323a1f4482d8704bf20594d13a5 ({username='slave'} at 127.21.84.135:48367)
I20240501 08:13:01.190436 27912 tablet_copy_service.cc:161] P f1525cbc82a24d13b65f136695c903af: Beginning new tablet copy session on tablet 9820951bffc749139229e2b79c8ab597 from peer 64998323a1f4482d8704bf20594d13a5 at {username='slave'} at 127.21.84.135:48367: session id = 64998323a1f4482d8704bf20594d13a5-9820951bffc749139229e2b79c8ab597
I20240501 08:13:01.194556 27912 tablet_copy_source_session.cc:205] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:01.196319 3730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9820951bffc749139229e2b79c8ab597. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:01.204584 30936 raft_consensus.cc:1203] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 08:13:01.208246 3730 tablet_copy_client.cc:783] T 9820951bffc749139229e2b79c8ab597 P 64998323a1f4482d8704bf20594d13a5: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:01.208613 3730 tablet_copy_client.cc:647] T 9820951bffc749139229e2b79c8ab597 P 64998323a1f4482d8704bf20594d13a5: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:01.211206 3730 tablet_copy_client.cc:523] T 9820951bffc749139229e2b79c8ab597 P 64998323a1f4482d8704bf20594d13a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:01.214907 28152 raft_consensus.cc:1203] T 00205a461b704a19bbd39bc71361e529 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 08:13:01.216564 3730 tablet_bootstrap.cc:492] T 9820951bffc749139229e2b79c8ab597 P 64998323a1f4482d8704bf20594d13a5: Bootstrap starting.
I20240501 08:13:01.222844 28020 consensus_queue.cc:237] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
W20240501 08:13:01.228497 27957 consensus_peers.cc:485] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 -> Peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): Couldn't send request to peer a29cd590999b4f32849ff5689cd53c11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5eccb9416d7d438893307f13dd4e73dc. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:01.229784 28152 raft_consensus.cc:1261] T 5eccb9416d7d438893307f13dd4e73dc P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:13:01.229962 30802 raft_consensus.cc:1261] T 5eccb9416d7d438893307f13dd4e73dc P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:13:01.231206 3723 consensus_queue.cc:1035] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:13:01.235198 3688 consensus_queue.cc:1035] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:13:01.235642 3543 raft_consensus.cc:2932] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:13:01.236029 28152 raft_consensus.cc:2932] T 5eccb9416d7d438893307f13dd4e73dc P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:13:01.251727 27616 catalog_manager.cc:5484] T 5eccb9416d7d438893307f13dd4e73dc P 91d11eae5ef6432ca86eb8ed446b6ea6 reported cstate change: config changed from index 6 to 7, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:13:01.255852 30802 raft_consensus.cc:2932] T 5eccb9416d7d438893307f13dd4e73dc P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } } }
I20240501 08:13:01.262105 3773 ts_tablet_manager.cc:902] T bc5778711a4c479881088af790f5b7d5 P 9a022e1a501f4d44a0468054d6c3de47: Initiating tablet copy from peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:13:01.283267 3764 ts_tablet_manager.cc:902] T f63215605ade41a5bc8d97a35ce33a31 P 91d11eae5ef6432ca86eb8ed446b6ea6: Initiating tablet copy from peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:13:01.284204 3795 tablet_bootstrap.cc:492] T 6c04880ea87d4894bd3a63d94f4a69f4 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:01.284592 3795 tablet_bootstrap.cc:492] T 6c04880ea87d4894bd3a63d94f4a69f4 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap complete.
I20240501 08:13:01.284852 3795 ts_tablet_manager.cc:1367] T 6c04880ea87d4894bd3a63d94f4a69f4 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.104s user 0.043s sys 0.013s
I20240501 08:13:01.285595 3795 raft_consensus.cc:348] T 6c04880ea87d4894bd3a63d94f4a69f4 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:13:01.285877 3795 raft_consensus.cc:729] T 6c04880ea87d4894bd3a63d94f4a69f4 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: LEARNER
I20240501 08:13:01.286105 3795 consensus_queue.cc:260] T 6c04880ea87d4894bd3a63d94f4a69f4 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:13:01.294662 30802 raft_consensus.cc:1203] T 4b3e3d39f82549079293d98357884bf0 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 08:13:01.302608 3795 ts_tablet_manager.cc:1398] T 6c04880ea87d4894bd3a63d94f4a69f4 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.018s user 0.008s sys 0.010s
I20240501 08:13:01.303375 28042 tablet_copy_service.cc:342] P aad81047009a4ffcbc450b60030583b5: Request end of tablet copy session 9a022e1a501f4d44a0468054d6c3de47-6c04880ea87d4894bd3a63d94f4a69f4 received from {username='slave'} at 127.21.84.134:40301
I20240501 08:13:01.303501 28042 tablet_copy_service.cc:433] P aad81047009a4ffcbc450b60030583b5: ending tablet copy session 9a022e1a501f4d44a0468054d6c3de47-6c04880ea87d4894bd3a63d94f4a69f4 on tablet 6c04880ea87d4894bd3a63d94f4a69f4 with peer 9a022e1a501f4d44a0468054d6c3de47
I20240501 08:13:01.311873 3773 tablet_copy_client.cc:308] T bc5778711a4c479881088af790f5b7d5 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.132:41323
I20240501 08:13:01.312500 28171 tablet_copy_service.cc:140] P 91d11eae5ef6432ca86eb8ed446b6ea6: Received BeginTabletCopySession request for tablet bc5778711a4c479881088af790f5b7d5 from peer 9a022e1a501f4d44a0468054d6c3de47 ({username='slave'} at 127.21.84.134:40845)
I20240501 08:13:01.312662 28171 tablet_copy_service.cc:161] P 91d11eae5ef6432ca86eb8ed446b6ea6: Beginning new tablet copy session on tablet bc5778711a4c479881088af790f5b7d5 from peer 9a022e1a501f4d44a0468054d6c3de47 at {username='slave'} at 127.21.84.134:40845: session id = 9a022e1a501f4d44a0468054d6c3de47-bc5778711a4c479881088af790f5b7d5
I20240501 08:13:01.315438 28171 tablet_copy_source_session.cc:205] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:01.316671 3773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc5778711a4c479881088af790f5b7d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:01.323099 27758 consensus_queue.cc:237] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
I20240501 08:13:01.334437 3730 tablet_bootstrap.cc:492] T 9820951bffc749139229e2b79c8ab597 P 64998323a1f4482d8704bf20594d13a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:01.329910 3773 tablet_copy_client.cc:783] T bc5778711a4c479881088af790f5b7d5 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:01.334846 3773 tablet_copy_client.cc:647] T bc5778711a4c479881088af790f5b7d5 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:01.337217 3773 tablet_copy_client.cc:523] T bc5778711a4c479881088af790f5b7d5 P 9a022e1a501f4d44a0468054d6c3de47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:01.341344 3773 tablet_bootstrap.cc:492] T bc5778711a4c479881088af790f5b7d5 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap starting.
I20240501 08:13:01.341416 27889 raft_consensus.cc:1261] T 32a54b70dcc447828ec293a9ff0afb80 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:13:01.340646 30802 raft_consensus.cc:1261] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:13:01.342203 3246 consensus_queue.cc:1035] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:13:01.342602 3655 consensus_queue.cc:1035] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 08:13:01.344754 27697 consensus_peers.cc:485] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 -> Peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): Couldn't send request to peer 31811708c6774037b82df44024b229b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32a54b70dcc447828ec293a9ff0afb80. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:01.350332 3730 tablet_bootstrap.cc:492] T 9820951bffc749139229e2b79c8ab597 P 64998323a1f4482d8704bf20594d13a5: Bootstrap complete.
I20240501 08:13:01.351910 3730 ts_tablet_manager.cc:1367] T 9820951bffc749139229e2b79c8ab597 P 64998323a1f4482d8704bf20594d13a5: Time spent bootstrapping tablet: real 0.135s user 0.045s sys 0.010s
I20240501 08:13:01.358628 3730 raft_consensus.cc:348] T 9820951bffc749139229e2b79c8ab597 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
I20240501 08:13:01.360074 3730 raft_consensus.cc:729] T 9820951bffc749139229e2b79c8ab597 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Initialized, Role: LEARNER
I20240501 08:13:01.360550 3730 consensus_queue.cc:260] T 9820951bffc749139229e2b79c8ab597 P 64998323a1f4482d8704bf20594d13a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: NON_VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: true } }
I20240501 08:13:01.363010 3730 ts_tablet_manager.cc:1398] T 9820951bffc749139229e2b79c8ab597 P 64998323a1f4482d8704bf20594d13a5: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240501 08:13:01.364284 27912 tablet_copy_service.cc:342] P f1525cbc82a24d13b65f136695c903af: Request end of tablet copy session 64998323a1f4482d8704bf20594d13a5-9820951bffc749139229e2b79c8ab597 received from {username='slave'} at 127.21.84.135:48367
I20240501 08:13:01.364405 27912 tablet_copy_service.cc:433] P f1525cbc82a24d13b65f136695c903af: ending tablet copy session 64998323a1f4482d8704bf20594d13a5-9820951bffc749139229e2b79c8ab597 on tablet 9820951bffc749139229e2b79c8ab597 with peer 64998323a1f4482d8704bf20594d13a5
I20240501 08:13:01.372553 3655 raft_consensus.cc:2932] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:13:01.377030 30802 raft_consensus.cc:2932] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:13:01.386588 3833 ts_tablet_manager.cc:902] T b6b0bcb4d07e4125bccf1711a094576e P 6331f77dae9d452184e8cd0cbf5f1b43: Initiating tablet copy from peer a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:13:01.385674 27615 catalog_manager.cc:5484] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index 6 to 7, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:01.411679 30669 raft_consensus.cc:1203] T b12559fc416a4f3cbffa018611226bcc P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 08:13:01.440534 27892 raft_consensus.cc:2932] T 32a54b70dcc447828ec293a9ff0afb80 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } }
I20240501 08:13:01.468641 3773 tablet_bootstrap.cc:492] T bc5778711a4c479881088af790f5b7d5 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:01.469074 3773 tablet_bootstrap.cc:492] T bc5778711a4c479881088af790f5b7d5 P 9a022e1a501f4d44a0468054d6c3de47: Bootstrap complete.
I20240501 08:13:01.469314 3773 ts_tablet_manager.cc:1367] T bc5778711a4c479881088af790f5b7d5 P 9a022e1a501f4d44a0468054d6c3de47: Time spent bootstrapping tablet: real 0.128s user 0.049s sys 0.018s
I20240501 08:13:01.474283 3764 tablet_copy_client.cc:308] T f63215605ade41a5bc8d97a35ce33a31 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.136:46845
I20240501 08:13:01.475020 30957 tablet_copy_service.cc:140] P a29cd590999b4f32849ff5689cd53c11: Received BeginTabletCopySession request for tablet f63215605ade41a5bc8d97a35ce33a31 from peer 91d11eae5ef6432ca86eb8ed446b6ea6 ({username='slave'} at 127.21.84.132:58509)
I20240501 08:13:01.475183 30957 tablet_copy_service.cc:161] P a29cd590999b4f32849ff5689cd53c11: Beginning new tablet copy session on tablet f63215605ade41a5bc8d97a35ce33a31 from peer 91d11eae5ef6432ca86eb8ed446b6ea6 at {username='slave'} at 127.21.84.132:58509: session id = 91d11eae5ef6432ca86eb8ed446b6ea6-f63215605ade41a5bc8d97a35ce33a31
I20240501 08:13:01.477124 3773 raft_consensus.cc:348] T bc5778711a4c479881088af790f5b7d5 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:13:01.477373 3773 raft_consensus.cc:729] T bc5778711a4c479881088af790f5b7d5 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Initialized, Role: LEARNER
I20240501 08:13:01.477646 3773 consensus_queue.cc:260] T bc5778711a4c479881088af790f5b7d5 P 9a022e1a501f4d44a0468054d6c3de47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: NON_VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: true } }
I20240501 08:13:01.478289 3773 ts_tablet_manager.cc:1398] T bc5778711a4c479881088af790f5b7d5 P 9a022e1a501f4d44a0468054d6c3de47: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20240501 08:13:01.481163 28171 tablet_copy_service.cc:342] P 91d11eae5ef6432ca86eb8ed446b6ea6: Request end of tablet copy session 9a022e1a501f4d44a0468054d6c3de47-bc5778711a4c479881088af790f5b7d5 received from {username='slave'} at 127.21.84.134:40845
I20240501 08:13:01.481304 28171 tablet_copy_service.cc:433] P 91d11eae5ef6432ca86eb8ed446b6ea6: ending tablet copy session 9a022e1a501f4d44a0468054d6c3de47-bc5778711a4c479881088af790f5b7d5 on tablet bc5778711a4c479881088af790f5b7d5 with peer 9a022e1a501f4d44a0468054d6c3de47
I20240501 08:13:01.481537 30957 tablet_copy_source_session.cc:205] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:01.483263 3764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63215605ade41a5bc8d97a35ce33a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:01.506033 3833 tablet_copy_client.cc:308] T b6b0bcb4d07e4125bccf1711a094576e P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.136:46845
I20240501 08:13:01.506768 30957 tablet_copy_service.cc:140] P a29cd590999b4f32849ff5689cd53c11: Received BeginTabletCopySession request for tablet b6b0bcb4d07e4125bccf1711a094576e from peer 6331f77dae9d452184e8cd0cbf5f1b43 ({username='slave'} at 127.21.84.129:48739)
I20240501 08:13:01.506965 30957 tablet_copy_service.cc:161] P a29cd590999b4f32849ff5689cd53c11: Beginning new tablet copy session on tablet b6b0bcb4d07e4125bccf1711a094576e from peer 6331f77dae9d452184e8cd0cbf5f1b43 at {username='slave'} at 127.21.84.129:48739: session id = 6331f77dae9d452184e8cd0cbf5f1b43-b6b0bcb4d07e4125bccf1711a094576e
I20240501 08:13:01.509518 30957 tablet_copy_source_session.cc:205] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:01.510850 3833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6b0bcb4d07e4125bccf1711a094576e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:01.519902 3764 tablet_copy_client.cc:783] T f63215605ade41a5bc8d97a35ce33a31 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:01.520277 3764 tablet_copy_client.cc:647] T f63215605ade41a5bc8d97a35ce33a31 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:01.530568 30537 raft_consensus.cc:1203] T e6fb30f10c754d3a8e73e124ec59768b P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 08:13:01.537987 3764 tablet_copy_client.cc:523] T f63215605ade41a5bc8d97a35ce33a31 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:01.541529 3764 tablet_bootstrap.cc:492] T f63215605ade41a5bc8d97a35ce33a31 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:13:01.547123 3833 tablet_copy_client.cc:783] T b6b0bcb4d07e4125bccf1711a094576e P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:01.547629 3833 tablet_copy_client.cc:647] T b6b0bcb4d07e4125bccf1711a094576e P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:01.547960 3859 ts_tablet_manager.cc:902] T 0e06cf862c954e8fb1ca087ec5184497 P a29cd590999b4f32849ff5689cd53c11: Initiating tablet copy from peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:13:01.549443 3833 tablet_copy_client.cc:523] T b6b0bcb4d07e4125bccf1711a094576e P 6331f77dae9d452184e8cd0cbf5f1b43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:01.552544 3833 tablet_bootstrap.cc:492] T b6b0bcb4d07e4125bccf1711a094576e P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap starting.
I20240501 08:13:01.567243 3833 tablet_bootstrap.cc:492] T b6b0bcb4d07e4125bccf1711a094576e P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:01.567718 3833 tablet_bootstrap.cc:492] T b6b0bcb4d07e4125bccf1711a094576e P 6331f77dae9d452184e8cd0cbf5f1b43: Bootstrap complete.
I20240501 08:13:01.568046 3833 ts_tablet_manager.cc:1367] T b6b0bcb4d07e4125bccf1711a094576e P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.008s
I20240501 08:13:01.568737 3833 raft_consensus.cc:348] T b6b0bcb4d07e4125bccf1711a094576e P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:13:01.569089 3833 raft_consensus.cc:729] T b6b0bcb4d07e4125bccf1711a094576e P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Initialized, Role: LEARNER
I20240501 08:13:01.569343 3833 consensus_queue.cc:260] T b6b0bcb4d07e4125bccf1711a094576e P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: NON_VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: true } }
I20240501 08:13:01.571326 3833 ts_tablet_manager.cc:1398] T b6b0bcb4d07e4125bccf1711a094576e P 6331f77dae9d452184e8cd0cbf5f1b43: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 08:13:01.572470 30957 tablet_copy_service.cc:342] P a29cd590999b4f32849ff5689cd53c11: Request end of tablet copy session 6331f77dae9d452184e8cd0cbf5f1b43-b6b0bcb4d07e4125bccf1711a094576e received from {username='slave'} at 127.21.84.129:48739
I20240501 08:13:01.572613 30957 tablet_copy_service.cc:433] P a29cd590999b4f32849ff5689cd53c11: ending tablet copy session 6331f77dae9d452184e8cd0cbf5f1b43-b6b0bcb4d07e4125bccf1711a094576e on tablet b6b0bcb4d07e4125bccf1711a094576e with peer 6331f77dae9d452184e8cd0cbf5f1b43
I20240501 08:13:01.581835 3859 tablet_copy_client.cc:308] T 0e06cf862c954e8fb1ca087ec5184497 P a29cd590999b4f32849ff5689cd53c11: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.130:36503
I20240501 08:13:01.584219 3764 tablet_bootstrap.cc:492] T f63215605ade41a5bc8d97a35ce33a31 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:01.584784 27912 tablet_copy_service.cc:140] P f1525cbc82a24d13b65f136695c903af: Received BeginTabletCopySession request for tablet 0e06cf862c954e8fb1ca087ec5184497 from peer a29cd590999b4f32849ff5689cd53c11 ({username='slave'} at 127.21.84.136:54747)
I20240501 08:13:01.584944 27912 tablet_copy_service.cc:161] P f1525cbc82a24d13b65f136695c903af: Beginning new tablet copy session on tablet 0e06cf862c954e8fb1ca087ec5184497 from peer a29cd590999b4f32849ff5689cd53c11 at {username='slave'} at 127.21.84.136:54747: session id = a29cd590999b4f32849ff5689cd53c11-0e06cf862c954e8fb1ca087ec5184497
I20240501 08:13:01.587785 27912 tablet_copy_source_session.cc:205] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:01.589545 3859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e06cf862c954e8fb1ca087ec5184497. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:01.590181 3764 tablet_bootstrap.cc:492] T f63215605ade41a5bc8d97a35ce33a31 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap complete.
I20240501 08:13:01.590451 3764 ts_tablet_manager.cc:1367] T f63215605ade41a5bc8d97a35ce33a31 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.049s user 0.011s sys 0.000s
I20240501 08:13:01.591142 3764 raft_consensus.cc:348] T f63215605ade41a5bc8d97a35ce33a31 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:13:01.591372 3764 raft_consensus.cc:729] T f63215605ade41a5bc8d97a35ce33a31 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: LEARNER
I20240501 08:13:01.591533 3764 consensus_queue.cc:260] T f63215605ade41a5bc8d97a35ce33a31 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:13:01.592042 3764 ts_tablet_manager.cc:1398] T f63215605ade41a5bc8d97a35ce33a31 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:13:01.602072 30957 tablet_copy_service.cc:342] P a29cd590999b4f32849ff5689cd53c11: Request end of tablet copy session 91d11eae5ef6432ca86eb8ed446b6ea6-f63215605ade41a5bc8d97a35ce33a31 received from {username='slave'} at 127.21.84.132:58509
I20240501 08:13:01.602234 30957 tablet_copy_service.cc:433] P a29cd590999b4f32849ff5689cd53c11: ending tablet copy session 91d11eae5ef6432ca86eb8ed446b6ea6-f63215605ade41a5bc8d97a35ce33a31 on tablet f63215605ade41a5bc8d97a35ce33a31 with peer 91d11eae5ef6432ca86eb8ed446b6ea6
I20240501 08:13:01.606751 3859 tablet_copy_client.cc:783] T 0e06cf862c954e8fb1ca087ec5184497 P a29cd590999b4f32849ff5689cd53c11: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:01.607151 3859 tablet_copy_client.cc:647] T 0e06cf862c954e8fb1ca087ec5184497 P a29cd590999b4f32849ff5689cd53c11: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:01.615389 3859 tablet_copy_client.cc:523] T 0e06cf862c954e8fb1ca087ec5184497 P a29cd590999b4f32849ff5689cd53c11: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:01.620901 3859 tablet_bootstrap.cc:492] T 0e06cf862c954e8fb1ca087ec5184497 P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:13:01.636873 30669 raft_consensus.cc:1203] T bc5778711a4c479881088af790f5b7d5 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 08:13:01.648463 3843 raft_consensus.cc:1050] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5: attempting to promote NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 to VOTER
I20240501 08:13:01.649654 3843 consensus_queue.cc:237] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:13:01.654027 30669 raft_consensus.cc:1203] T 6c04880ea87d4894bd3a63d94f4a69f4 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 08:13:01.661988 30802 raft_consensus.cc:1203] T 9820951bffc749139229e2b79c8ab597 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 08:13:01.668123 3859 tablet_bootstrap.cc:492] T 0e06cf862c954e8fb1ca087ec5184497 P a29cd590999b4f32849ff5689cd53c11: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:01.669672 3859 tablet_bootstrap.cc:492] T 0e06cf862c954e8fb1ca087ec5184497 P a29cd590999b4f32849ff5689cd53c11: Bootstrap complete.
I20240501 08:13:01.671029 27758 raft_consensus.cc:1261] T f607d2e086cc4352a05a5ef2b02788cc P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:13:01.676395 3800 consensus_queue.cc:1035] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:13:01.679510 3859 ts_tablet_manager.cc:1367] T 0e06cf862c954e8fb1ca087ec5184497 P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.059s user 0.017s sys 0.001s
I20240501 08:13:01.681270 30934 raft_consensus.cc:1261] T f607d2e086cc4352a05a5ef2b02788cc P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:13:01.681082 3859 raft_consensus.cc:348] T 0e06cf862c954e8fb1ca087ec5184497 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:13:01.681479 3859 raft_consensus.cc:729] T 0e06cf862c954e8fb1ca087ec5184497 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: LEARNER
I20240501 08:13:01.682287 3843 consensus_queue.cc:1035] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:13:01.682533 27889 raft_consensus.cc:1261] T f607d2e086cc4352a05a5ef2b02788cc P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:13:01.683476 3800 consensus_queue.cc:1035] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:13:01.689041 3859 consensus_queue.cc:260] T 0e06cf862c954e8fb1ca087ec5184497 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:13:01.690558 3859 ts_tablet_manager.cc:1398] T 0e06cf862c954e8fb1ca087ec5184497 P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 08:13:01.693913 28152 raft_consensus.cc:1203] T f63215605ade41a5bc8d97a35ce33a31 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 08:13:01.694324 27912 tablet_copy_service.cc:342] P f1525cbc82a24d13b65f136695c903af: Request end of tablet copy session a29cd590999b4f32849ff5689cd53c11-0e06cf862c954e8fb1ca087ec5184497 received from {username='slave'} at 127.21.84.136:54747
I20240501 08:13:01.694427 27912 tablet_copy_service.cc:433] P f1525cbc82a24d13b65f136695c903af: ending tablet copy session a29cd590999b4f32849ff5689cd53c11-0e06cf862c954e8fb1ca087ec5184497 on tablet 0e06cf862c954e8fb1ca087ec5184497 with peer a29cd590999b4f32849ff5689cd53c11
I20240501 08:13:01.699059 3859 ts_tablet_manager.cc:902] T 5eccb9416d7d438893307f13dd4e73dc P a29cd590999b4f32849ff5689cd53c11: Initiating tablet copy from peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:13:01.726719 3843 raft_consensus.cc:1050] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5: attempting to promote NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 to VOTER
I20240501 08:13:01.727682 3843 consensus_queue.cc:237] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:13:01.729228 3632 raft_consensus.cc:2932] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:13:01.729749 27889 raft_consensus.cc:2932] T f607d2e086cc4352a05a5ef2b02788cc P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:13:01.734402 27758 raft_consensus.cc:2932] T f607d2e086cc4352a05a5ef2b02788cc P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:13:01.738113 27616 catalog_manager.cc:5484] T f607d2e086cc4352a05a5ef2b02788cc P f1525cbc82a24d13b65f136695c903af reported cstate change: config changed from index 5 to 6, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:13:01.744503 30934 raft_consensus.cc:2932] T f607d2e086cc4352a05a5ef2b02788cc P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:13:01.745893 30802 consensus_queue.cc:237] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:13:01.748672 27892 raft_consensus.cc:1261] T f607d2e086cc4352a05a5ef2b02788cc P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:13:01.749073 27760 raft_consensus.cc:1261] T 00205a461b704a19bbd39bc71361e529 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:13:01.750313 3800 consensus_queue.cc:1035] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:13:01.750802 3800 consensus_queue.cc:1035] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:13:01.755194 27758 raft_consensus.cc:1261] T f607d2e086cc4352a05a5ef2b02788cc P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:13:01.755852 3881 consensus_queue.cc:1035] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:13:01.759290 3632 raft_consensus.cc:2932] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:13:01.762387 27761 raft_consensus.cc:2932] T f607d2e086cc4352a05a5ef2b02788cc P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:13:01.766983 30669 raft_consensus.cc:1261] T 00205a461b704a19bbd39bc71361e529 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:13:01.767369 28152 raft_consensus.cc:1261] T 00205a461b704a19bbd39bc71361e529 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:13:01.767870 3800 consensus_queue.cc:1035] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:13:01.770138 3883 consensus_queue.cc:1035] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:13:01.775557 27619 catalog_manager.cc:5484] T f607d2e086cc4352a05a5ef2b02788cc P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index 6 to 7, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:13:01.786795 30916 tablet_service.cc:1514] Processing DeleteTablet for tablet f607d2e086cc4352a05a5ef2b02788cc with delete_type TABLET_DATA_TOMBSTONED (TS a29cd590999b4f32849ff5689cd53c11 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54036
I20240501 08:13:01.787818 3531 raft_consensus.cc:1050] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af: attempting to promote NON_VOTER a29cd590999b4f32849ff5689cd53c11 to VOTER
I20240501 08:13:01.788712 3531 consensus_queue.cc:237] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:13:01.789844 27605 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f607d2e086cc4352a05a5ef2b02788cc with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:13:01.790736 27892 raft_consensus.cc:2932] T f607d2e086cc4352a05a5ef2b02788cc P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:13:01.799088 3246 raft_consensus.cc:1050] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43: attempting to promote NON_VOTER 31811708c6774037b82df44024b229b6 to VOTER
I20240501 08:13:01.803679 3246 consensus_queue.cc:237] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } }
I20240501 08:13:01.806214 3881 raft_consensus.cc:2932] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:13:01.809383 27760 raft_consensus.cc:2932] T 00205a461b704a19bbd39bc71361e529 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:13:01.819108 27616 catalog_manager.cc:5484] T 00205a461b704a19bbd39bc71361e529 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index 5 to 6, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:13:01.821593 28151 raft_consensus.cc:2932] T 00205a461b704a19bbd39bc71361e529 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:13:01.829721 30802 consensus_queue.cc:237] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:13:01.834152 27760 raft_consensus.cc:1261] T 00205a461b704a19bbd39bc71361e529 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:13:01.834344 30935 raft_consensus.cc:1261] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:13:01.834877 3881 consensus_queue.cc:1035] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:13:01.835458 3859 tablet_copy_client.cc:308] T 5eccb9416d7d438893307f13dd4e73dc P a29cd590999b4f32849ff5689cd53c11: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.131:36305
I20240501 08:13:01.836269 30537 raft_consensus.cc:1261] T 7f7211c8c0e041bf89930aa50a4438ea P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:13:01.836758 28042 tablet_copy_service.cc:140] P aad81047009a4ffcbc450b60030583b5: Received BeginTabletCopySession request for tablet 5eccb9416d7d438893307f13dd4e73dc from peer a29cd590999b4f32849ff5689cd53c11 ({username='slave'} at 127.21.84.136:51325)
I20240501 08:13:01.836975 28042 tablet_copy_service.cc:161] P aad81047009a4ffcbc450b60030583b5: Beginning new tablet copy session on tablet 5eccb9416d7d438893307f13dd4e73dc from peer a29cd590999b4f32849ff5689cd53c11 at {username='slave'} at 127.21.84.136:51325: session id = a29cd590999b4f32849ff5689cd53c11-5eccb9416d7d438893307f13dd4e73dc
I20240501 08:13:01.838429 3843 raft_consensus.cc:2932] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:13:01.840039 28042 tablet_copy_source_session.cc:205] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:01.841604 3859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eccb9416d7d438893307f13dd4e73dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:01.842392 27760 raft_consensus.cc:2932] T 00205a461b704a19bbd39bc71361e529 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:13:01.843410 28152 raft_consensus.cc:1261] T 00205a461b704a19bbd39bc71361e529 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:13:01.844516 2620 consensus_queue.cc:1035] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:13:01.848224 2620 consensus_queue.cc:1035] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:13:01.849228 3859 tablet_copy_client.cc:783] T 5eccb9416d7d438893307f13dd4e73dc P a29cd590999b4f32849ff5689cd53c11: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:01.849195 3883 consensus_queue.cc:1035] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:13:01.849628 3859 tablet_copy_client.cc:647] T 5eccb9416d7d438893307f13dd4e73dc P a29cd590999b4f32849ff5689cd53c11: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:01.852391 28152 raft_consensus.cc:2932] T 00205a461b704a19bbd39bc71361e529 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:13:01.854594 3664 raft_consensus.cc:1050] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6: attempting to promote NON_VOTER 64998323a1f4482d8704bf20594d13a5 to VOTER
I20240501 08:13:01.855448 3664 consensus_queue.cc:237] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:13:01.859742 27605 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 00205a461b704a19bbd39bc71361e529 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:13:01.864068 27760 raft_consensus.cc:1261] T 7f7211c8c0e041bf89930aa50a4438ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:13:01.865233 30670 raft_consensus.cc:1261] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:13:01.865187 27619 catalog_manager.cc:5484] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index 6 to 7, VOTER 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) evicted. New cstate: current_term: 1 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:01.865567 2620 consensus_queue.cc:1035] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:13:01.866691 3887 tablet_replica.cc:330] T f607d2e086cc4352a05a5ef2b02788cc P a29cd590999b4f32849ff5689cd53c11: stopping tablet replica
I20240501 08:13:01.866770 3859 tablet_copy_client.cc:523] T 5eccb9416d7d438893307f13dd4e73dc P a29cd590999b4f32849ff5689cd53c11: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:01.867062 3887 raft_consensus.cc:2229] T f607d2e086cc4352a05a5ef2b02788cc P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:01.867558 3887 raft_consensus.cc:2258] T f607d2e086cc4352a05a5ef2b02788cc P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:01.867667 27892 raft_consensus.cc:1261] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:13:01.868796 3887 ts_tablet_manager.cc:1875] T f607d2e086cc4352a05a5ef2b02788cc P a29cd590999b4f32849ff5689cd53c11: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:01.870354 3654 consensus_queue.cc:1035] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:13:01.870688 3655 consensus_queue.cc:1035] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:13:01.870937 3859 tablet_bootstrap.cc:492] T 5eccb9416d7d438893307f13dd4e73dc P a29cd590999b4f32849ff5689cd53c11: Bootstrap starting.
I20240501 08:13:01.874050 30650 tablet_service.cc:1514] Processing DeleteTablet for tablet 00205a461b704a19bbd39bc71361e529 with delete_type TABLET_DATA_TOMBSTONED (TS 9a022e1a501f4d44a0468054d6c3de47 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57156
I20240501 08:13:01.875793 30802 raft_consensus.cc:1261] T 4b3e3d39f82549079293d98357884bf0 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:13:01.876120 30934 raft_consensus.cc:1261] T 4b3e3d39f82549079293d98357884bf0 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:13:01.876467 28022 raft_consensus.cc:1261] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:13:01.877418 3354 consensus_queue.cc:1035] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:13:01.877990 3475 consensus_queue.cc:1035] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:13:01.886077 3475 consensus_queue.cc:1035] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:13:01.890030 3903 tablet_replica.cc:330] T 00205a461b704a19bbd39bc71361e529 P 9a022e1a501f4d44a0468054d6c3de47: stopping tablet replica
I20240501 08:13:01.894075 3903 raft_consensus.cc:2229] T 00205a461b704a19bbd39bc71361e529 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:01.894695 3903 pending_rounds.cc:62] T 00205a461b704a19bbd39bc71361e529 P 9a022e1a501f4d44a0468054d6c3de47: Trying to abort 1 pending ops.
I20240501 08:13:01.894838 3903 pending_rounds.cc:68] T 00205a461b704a19bbd39bc71361e529 P 9a022e1a501f4d44a0468054d6c3de47: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7022801640354332672 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "00205a461b704a19bbd39bc71361e529" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } } }
I20240501 08:13:01.895505 3903 raft_consensus.cc:2894] T 00205a461b704a19bbd39bc71361e529 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20240501 08:13:01.895665 3903 raft_consensus.cc:2866] T 00205a461b704a19bbd39bc71361e529 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 08:13:01.895784 3903 raft_consensus.cc:2258] T 00205a461b704a19bbd39bc71361e529 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:01.901088 2620 raft_consensus.cc:2932] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:13:01.905277 3816 raft_consensus.cc:1050] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5: attempting to promote NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 to VOTER
I20240501 08:13:01.906380 3816 consensus_queue.cc:237] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:13:01.907732 30936 raft_consensus.cc:2932] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:13:01.913888 30537 raft_consensus.cc:1261] T 802538381de843ab8d326014a0b4ec15 P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:13:01.916415 3654 consensus_queue.cc:1035] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:13:01.917204 27616 catalog_manager.cc:5484] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index 7 to 8, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:13:01.923355 30670 raft_consensus.cc:1261] T b12559fc416a4f3cbffa018611226bcc P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:13:01.924038 3543 consensus_queue.cc:1035] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:13:01.925532 27761 raft_consensus.cc:1203] T b6b0bcb4d07e4125bccf1711a094576e P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:01.931723 3903 ts_tablet_manager.cc:1875] T 00205a461b704a19bbd39bc71361e529 P 9a022e1a501f4d44a0468054d6c3de47: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:01.943786 30936 raft_consensus.cc:1261] T b12559fc416a4f3cbffa018611226bcc P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:13:01.944479 27889 raft_consensus.cc:1261] T b12559fc416a4f3cbffa018611226bcc P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:13:01.945973 3543 consensus_queue.cc:1035] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:13:01.946635 3543 consensus_queue.cc:1035] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:13:01.947981 3903 ts_tablet_manager.cc:1888] T 00205a461b704a19bbd39bc71361e529 P 9a022e1a501f4d44a0468054d6c3de47: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 08:13:01.948341 3903 log.cc:1196] T 00205a461b704a19bbd39bc71361e529 P 9a022e1a501f4d44a0468054d6c3de47: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/wals/00205a461b704a19bbd39bc71361e529
I20240501 08:13:01.952638 27604 catalog_manager.cc:4830] TS 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): tablet 00205a461b704a19bbd39bc71361e529 (table rebalancer_extra_table_60 [id=9a256e3f732a4230901aea19c32440ba]) successfully deleted
I20240501 08:13:01.956998 3723 raft_consensus.cc:1050] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5: attempting to promote NON_VOTER 31811708c6774037b82df44024b229b6 to VOTER
I20240501 08:13:01.957926 3723 consensus_queue.cc:237] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } }
I20240501 08:13:01.967056 3909 ts_tablet_manager.cc:902] T 32a54b70dcc447828ec293a9ff0afb80 P 31811708c6774037b82df44024b229b6: Initiating tablet copy from peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:13:01.972334 3543 raft_consensus.cc:2932] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:13:01.979974 30936 raft_consensus.cc:2932] T b12559fc416a4f3cbffa018611226bcc P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:13:01.988963 3859 tablet_bootstrap.cc:492] T 5eccb9416d7d438893307f13dd4e73dc P a29cd590999b4f32849ff5689cd53c11: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:01.989974 3859 tablet_bootstrap.cc:492] T 5eccb9416d7d438893307f13dd4e73dc P a29cd590999b4f32849ff5689cd53c11: Bootstrap complete.
I20240501 08:13:01.990309 3859 ts_tablet_manager.cc:1367] T 5eccb9416d7d438893307f13dd4e73dc P a29cd590999b4f32849ff5689cd53c11: Time spent bootstrapping tablet: real 0.119s user 0.060s sys 0.007s
I20240501 08:13:01.995347 27889 raft_consensus.cc:2932] T b12559fc416a4f3cbffa018611226bcc P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:13:01.992928 3859 raft_consensus.cc:348] T 5eccb9416d7d438893307f13dd4e73dc P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:13:02.001127 3859 raft_consensus.cc:729] T 5eccb9416d7d438893307f13dd4e73dc P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Initialized, Role: LEARNER
I20240501 08:13:02.002060 3859 consensus_queue.cc:260] T 5eccb9416d7d438893307f13dd4e73dc P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: NON_VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: true } }
I20240501 08:13:02.005524 3859 ts_tablet_manager.cc:1398] T 5eccb9416d7d438893307f13dd4e73dc P a29cd590999b4f32849ff5689cd53c11: Time spent starting tablet: real 0.015s user 0.000s sys 0.003s
I20240501 08:13:02.006335 30670 raft_consensus.cc:2932] T b12559fc416a4f3cbffa018611226bcc P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:13:02.006493 27616 catalog_manager.cc:5484] T b12559fc416a4f3cbffa018611226bcc P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index 7 to 8, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:13:02.007778 28042 tablet_copy_service.cc:342] P aad81047009a4ffcbc450b60030583b5: Request end of tablet copy session a29cd590999b4f32849ff5689cd53c11-5eccb9416d7d438893307f13dd4e73dc received from {username='slave'} at 127.21.84.136:51325
I20240501 08:13:02.007915 28042 tablet_copy_service.cc:433] P aad81047009a4ffcbc450b60030583b5: ending tablet copy session a29cd590999b4f32849ff5689cd53c11-5eccb9416d7d438893307f13dd4e73dc on tablet 5eccb9416d7d438893307f13dd4e73dc with peer a29cd590999b4f32849ff5689cd53c11
I20240501 08:13:02.015395 30537 raft_consensus.cc:2932] T 7f7211c8c0e041bf89930aa50a4438ea P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:13:02.021426 3909 tablet_copy_client.cc:308] T 32a54b70dcc447828ec293a9ff0afb80 P 31811708c6774037b82df44024b229b6: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.129:43471
I20240501 08:13:02.022401 27781 tablet_copy_service.cc:140] P 6331f77dae9d452184e8cd0cbf5f1b43: Received BeginTabletCopySession request for tablet 32a54b70dcc447828ec293a9ff0afb80 from peer 31811708c6774037b82df44024b229b6 ({username='slave'} at 127.21.84.133:35347)
I20240501 08:13:02.022631 27781 tablet_copy_service.cc:161] P 6331f77dae9d452184e8cd0cbf5f1b43: Beginning new tablet copy session on tablet 32a54b70dcc447828ec293a9ff0afb80 from peer 31811708c6774037b82df44024b229b6 at {username='slave'} at 127.21.84.133:35347: session id = 31811708c6774037b82df44024b229b6-32a54b70dcc447828ec293a9ff0afb80
I20240501 08:13:02.026664 27781 tablet_copy_source_session.cc:205] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:02.028815 3909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a54b70dcc447828ec293a9ff0afb80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:02.041534 30936 raft_consensus.cc:1203] T 5eccb9416d7d438893307f13dd4e73dc P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 08:13:02.044127 3475 raft_consensus.cc:1050] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6: attempting to promote NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 to VOTER
I20240501 08:13:02.045105 3475 consensus_queue.cc:237] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:13:02.053895 30536 raft_consensus.cc:1261] T e6fb30f10c754d3a8e73e124ec59768b P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:13:02.054646 3543 consensus_queue.cc:1035] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:13:02.057379 3900 raft_consensus.cc:2932] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:13:02.058943 30670 raft_consensus.cc:2932] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:13:02.061769 3909 tablet_copy_client.cc:783] T 32a54b70dcc447828ec293a9ff0afb80 P 31811708c6774037b82df44024b229b6: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:02.064113 3909 tablet_copy_client.cc:647] T 32a54b70dcc447828ec293a9ff0afb80 P 31811708c6774037b82df44024b229b6: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:02.064364 30534 raft_consensus.cc:2932] T 802538381de843ab8d326014a0b4ec15 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:13:02.066107 3909 tablet_copy_client.cc:523] T 32a54b70dcc447828ec293a9ff0afb80 P 31811708c6774037b82df44024b229b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:02.067401 27615 catalog_manager.cc:5484] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index 7 to 8, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:13:02.071249 30672 raft_consensus.cc:1261] T e6fb30f10c754d3a8e73e124ec59768b P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:13:02.072201 3543 consensus_queue.cc:1035] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:13:02.073828 27761 consensus_queue.cc:237] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } }
I20240501 08:13:02.081485 3887 ts_tablet_manager.cc:1888] T f607d2e086cc4352a05a5ef2b02788cc P a29cd590999b4f32849ff5689cd53c11: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 08:13:02.083777 30537 raft_consensus.cc:1261] T 802538381de843ab8d326014a0b4ec15 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:13:02.085242 3887 log.cc:1196] T f607d2e086cc4352a05a5ef2b02788cc P a29cd590999b4f32849ff5689cd53c11: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/wals/f607d2e086cc4352a05a5ef2b02788cc
I20240501 08:13:02.086143 27606 catalog_manager.cc:4830] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): tablet f607d2e086cc4352a05a5ef2b02788cc (table rebalancer_extra_table_22 [id=dbd065db73d1428c8fe77bd9af3302f6]) successfully deleted
I20240501 08:13:02.087422 3909 tablet_bootstrap.cc:492] T 32a54b70dcc447828ec293a9ff0afb80 P 31811708c6774037b82df44024b229b6: Bootstrap starting.
I20240501 08:13:02.087563 28016 consensus_queue.cc:237] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:13:02.089643 30935 raft_consensus.cc:1261] T b12559fc416a4f3cbffa018611226bcc P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:13:02.090169 30668 raft_consensus.cc:1261] T b12559fc416a4f3cbffa018611226bcc P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:13:02.090723 3688 consensus_queue.cc:1035] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:13:02.090973 3246 consensus_queue.cc:1035] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:13:02.091375 3688 raft_consensus.cc:1050] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5: attempting to promote NON_VOTER 9a022e1a501f4d44a0468054d6c3de47 to VOTER
I20240501 08:13:02.092151 3688 consensus_queue.cc:237] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:13:02.093967 3816 consensus_queue.cc:1035] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:13:02.094756 30670 raft_consensus.cc:1261] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:13:02.094758 3354 raft_consensus.cc:2932] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:13:02.095801 3246 consensus_queue.cc:1035] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:13:02.095661 30535 raft_consensus.cc:2932] T 802538381de843ab8d326014a0b4ec15 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:13:02.096889 3543 raft_consensus.cc:2932] T b12559fc416a4f3cbffa018611226bcc P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:13:02.097641 30935 raft_consensus.cc:2932] T b12559fc416a4f3cbffa018611226bcc P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:13:02.094646 3900 raft_consensus.cc:2932] T 802538381de843ab8d326014a0b4ec15 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:13:02.100205 28016 raft_consensus.cc:2932] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:13:02.104817 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 802538381de843ab8d326014a0b4ec15 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:13:02.105312 27616 catalog_manager.cc:5484] T b12559fc416a4f3cbffa018611226bcc P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index 8 to 9, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:13:02.106516 30937 raft_consensus.cc:2932] T 4b3e3d39f82549079293d98357884bf0 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:13:02.112767 27619 catalog_manager.cc:5484] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 7 to 8, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:13:02.117038 30668 raft_consensus.cc:2932] T b12559fc416a4f3cbffa018611226bcc P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:13:02.127272 30670 raft_consensus.cc:2932] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:13:02.128721 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b12559fc416a4f3cbffa018611226bcc with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:13:02.132490 30533 raft_consensus.cc:1261] T 6c04880ea87d4894bd3a63d94f4a69f4 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:13:02.132989 30668 raft_consensus.cc:1261] T 6c04880ea87d4894bd3a63d94f4a69f4 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:13:02.133148 3816 consensus_queue.cc:1035] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:13:02.135588 27612 catalog_manager.cc:5484] T 802538381de843ab8d326014a0b4ec15 P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index 8 to 9, VOTER f1525cbc82a24d13b65f136695c903af (127.21.84.130) evicted. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:13:02.143635 3816 consensus_queue.cc:1035] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:13:02.145861 27761 raft_consensus.cc:1261] T 6c04880ea87d4894bd3a63d94f4a69f4 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:13:02.150555 3688 consensus_queue.cc:1035] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:13:02.185492 3688 raft_consensus.cc:2932] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:13:02.186509 30670 raft_consensus.cc:2932] T e6fb30f10c754d3a8e73e124ec59768b P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:13:02.191170 3816 raft_consensus.cc:2932] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:13:02.193744 30668 raft_consensus.cc:2932] T 6c04880ea87d4894bd3a63d94f4a69f4 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:13:02.194612 30536 raft_consensus.cc:2932] T e6fb30f10c754d3a8e73e124ec59768b P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:13:02.197872 30533 raft_consensus.cc:2932] T 6c04880ea87d4894bd3a63d94f4a69f4 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:13:02.200294 27758 raft_consensus.cc:2932] T 7f7211c8c0e041bf89930aa50a4438ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:13:02.204586 27761 raft_consensus.cc:2932] T 6c04880ea87d4894bd3a63d94f4a69f4 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:13:02.206272 27616 catalog_manager.cc:5484] T e6fb30f10c754d3a8e73e124ec59768b P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index 7 to 8, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:13:02.215255 27616 catalog_manager.cc:5484] T 6c04880ea87d4894bd3a63d94f4a69f4 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 7 to 8, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:13:02.227909 3722 raft_consensus.cc:1050] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11: attempting to promote NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 to VOTER
I20240501 08:13:02.228777 3722 consensus_queue.cc:237] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:13:02.232421 27871 tablet_service.cc:1514] Processing DeleteTablet for tablet b12559fc416a4f3cbffa018611226bcc with delete_type TABLET_DATA_TOMBSTONED (TS f1525cbc82a24d13b65f136695c903af not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:48054
I20240501 08:13:02.233206 27889 consensus_queue.cc:237] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:13:02.240324 30533 raft_consensus.cc:1261] T 7f7211c8c0e041bf89930aa50a4438ea P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:13:02.241266 3531 consensus_queue.cc:1035] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:13:02.241890 3938 tablet_replica.cc:330] T b12559fc416a4f3cbffa018611226bcc P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:13:02.242838 30668 raft_consensus.cc:1261] T bc5778711a4c479881088af790f5b7d5 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 LEARNER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:13:02.243002 30803 raft_consensus.cc:2932] T 4b3e3d39f82549079293d98357884bf0 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:13:02.245630 3354 consensus_queue.cc:1035] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:13:02.246234 30535 raft_consensus.cc:1261] T bc5778711a4c479881088af790f5b7d5 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:13:02.246644 3938 raft_consensus.cc:2229] T b12559fc416a4f3cbffa018611226bcc P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:02.246863 3938 raft_consensus.cc:2258] T b12559fc416a4f3cbffa018611226bcc P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:02.246774 3930 consensus_queue.cc:1035] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:13:02.248059 30536 raft_consensus.cc:2932] T 7f7211c8c0e041bf89930aa50a4438ea P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:13:02.248795 28152 raft_consensus.cc:1261] T f63215605ade41a5bc8d97a35ce33a31 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:13:02.251127 3236 consensus_queue.cc:1035] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:13:02.253049 27761 raft_consensus.cc:1261] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:13:02.248235 3938 ts_tablet_manager.cc:1875] T b12559fc416a4f3cbffa018611226bcc P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:02.245648 3531 raft_consensus.cc:2932] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:13:02.254531 2620 raft_consensus.cc:1050] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af: attempting to promote NON_VOTER 64998323a1f4482d8704bf20594d13a5 to VOTER
I20240501 08:13:02.255263 3722 consensus_queue.cc:1035] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:13:02.255265 2620 consensus_queue.cc:237] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:13:02.256317 30534 raft_consensus.cc:1261] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:13:02.256930 3236 consensus_queue.cc:1035] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:13:02.266163 27616 catalog_manager.cc:5484] T 7f7211c8c0e041bf89930aa50a4438ea P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index 8 to 9, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:13:02.274102 27741 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f7211c8c0e041bf89930aa50a4438ea with delete_type TABLET_DATA_TOMBSTONED (TS 6331f77dae9d452184e8cd0cbf5f1b43 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:60422
I20240501 08:13:02.282265 3938 ts_tablet_manager.cc:1888] T b12559fc416a4f3cbffa018611226bcc P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 08:13:02.285032 3938 log.cc:1196] T b12559fc416a4f3cbffa018611226bcc P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/b12559fc416a4f3cbffa018611226bcc
I20240501 08:13:02.287195 3946 tablet_replica.cc:330] T 7f7211c8c0e041bf89930aa50a4438ea P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:13:02.294198 3946 raft_consensus.cc:2229] T 7f7211c8c0e041bf89930aa50a4438ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:02.295231 3946 raft_consensus.cc:2258] T 7f7211c8c0e041bf89930aa50a4438ea P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:02.295596 3236 raft_consensus.cc:2932] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:13:02.299518 27871 tablet_service.cc:1514] Processing DeleteTablet for tablet 802538381de843ab8d326014a0b4ec15 with delete_type TABLET_DATA_TOMBSTONED (TS f1525cbc82a24d13b65f136695c903af not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:48054
I20240501 08:13:02.300277 3938 tablet_replica.cc:330] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af: stopping tablet replica
I20240501 08:13:02.300479 3938 raft_consensus.cc:2229] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:02.300662 3938 pending_rounds.cc:62] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af: Trying to abort 1 pending ops.
I20240501 08:13:02.300761 3938 pending_rounds.cc:68] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7022801640650489856 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "802538381de843ab8d326014a0b4ec15" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { replace: true } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } } }
I20240501 08:13:02.301259 3938 raft_consensus.cc:2894] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240501 08:13:02.301414 3938 raft_consensus.cc:2866] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 08:13:02.301470 30670 raft_consensus.cc:1261] T 9820951bffc749139229e2b79c8ab597 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:13:02.301863 30803 raft_consensus.cc:1261] T 9820951bffc749139229e2b79c8ab597 P 64998323a1f4482d8704bf20594d13a5 [term 1 LEARNER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:13:02.301538 3938 raft_consensus.cc:2258] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:02.302213 3712 consensus_queue.cc:1035] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:13:02.302487 3913 consensus_queue.cc:1035] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:13:02.302534 3946 ts_tablet_manager.cc:1875] T 7f7211c8c0e041bf89930aa50a4438ea P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:02.302040 27761 raft_consensus.cc:2932] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:13:02.306847 28151 raft_consensus.cc:2932] T f63215605ade41a5bc8d97a35ce33a31 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:13:02.307457 27616 catalog_manager.cc:5484] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index 5 to 6, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:02.308318 3938 ts_tablet_manager.cc:1875] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:02.304461 30537 raft_consensus.cc:2932] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:13:02.323427 3946 ts_tablet_manager.cc:1888] T 7f7211c8c0e041bf89930aa50a4438ea P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 08:13:02.323559 3946 log.cc:1196] T 7f7211c8c0e041bf89930aa50a4438ea P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/7f7211c8c0e041bf89930aa50a4438ea
I20240501 08:13:02.324236 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet 7f7211c8c0e041bf89930aa50a4438ea (table rebalance_test_table_1 [id=0ee45bbb25734f82b00abd2da25300d1]) successfully deleted
I20240501 08:13:02.333611 28152 raft_consensus.cc:1261] T e6fb30f10c754d3a8e73e124ec59768b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:13:02.337752 28020 raft_consensus.cc:1261] T bc5778711a4c479881088af790f5b7d5 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:13:02.337795 3938 ts_tablet_manager.cc:1888] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 08:13:02.338948 3688 consensus_queue.cc:1035] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:13:02.337973 3938 log.cc:1196] T 802538381de843ab8d326014a0b4ec15 P f1525cbc82a24d13b65f136695c903af: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/wals/802538381de843ab8d326014a0b4ec15
I20240501 08:13:02.348169 28020 consensus_queue.cc:237] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } }
I20240501 08:13:02.348728 30934 raft_consensus.cc:1203] T 0e06cf862c954e8fb1ca087ec5184497 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 08:13:02.351099 30537 raft_consensus.cc:1261] T e6fb30f10c754d3a8e73e124ec59768b P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:13:02.351464 30672 raft_consensus.cc:1261] T e6fb30f10c754d3a8e73e124ec59768b P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:13:02.351771 3723 consensus_queue.cc:1035] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:13:02.352056 3816 consensus_queue.cc:1035] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:13:02.350385 28016 consensus_queue.cc:237] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:13:02.355336 30669 raft_consensus.cc:1261] T 6c04880ea87d4894bd3a63d94f4a69f4 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:13:02.355978 3723 consensus_queue.cc:1035] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:13:02.356611 3723 raft_consensus.cc:2932] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:13:02.360889 30669 raft_consensus.cc:2932] T e6fb30f10c754d3a8e73e124ec59768b P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:13:02.362402 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e6fb30f10c754d3a8e73e124ec59768b with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:13:02.363426 30533 raft_consensus.cc:1261] T 6c04880ea87d4894bd3a63d94f4a69f4 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:13:02.363919 3723 consensus_queue.cc:1035] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:13:02.368755 3913 raft_consensus.cc:2932] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:13:02.369532 27615 catalog_manager.cc:5484] T e6fb30f10c754d3a8e73e124ec59768b P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 8 to 9, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:02.373494 30803 raft_consensus.cc:2932] T 9820951bffc749139229e2b79c8ab597 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:13:02.374650 27607 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7f7211c8c0e041bf89930aa50a4438ea with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:13:02.374881 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet b12559fc416a4f3cbffa018611226bcc (table rebalance_test_table_17 [id=1121d4c7d5444060ac6df36744663e41]) successfully deleted
I20240501 08:13:02.375169 30934 raft_consensus.cc:1261] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:13:02.378799 3816 raft_consensus.cc:2932] T 6c04880ea87d4894bd3a63d94f4a69f4 P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:13:02.383062 30672 raft_consensus.cc:2932] T 6c04880ea87d4894bd3a63d94f4a69f4 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:13:02.384804 27612 catalog_manager.cc:5484] T 9820951bffc749139229e2b79c8ab597 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index 7 to 8, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:13:02.387503 28016 raft_consensus.cc:1261] T 9820951bffc749139229e2b79c8ab597 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:13:02.387256 3930 raft_consensus.cc:2932] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:13:02.389298 30533 raft_consensus.cc:2932] T bc5778711a4c479881088af790f5b7d5 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:13:02.390892 3909 tablet_bootstrap.cc:492] T 32a54b70dcc447828ec293a9ff0afb80 P 31811708c6774037b82df44024b229b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:02.392261 3909 tablet_bootstrap.cc:492] T 32a54b70dcc447828ec293a9ff0afb80 P 31811708c6774037b82df44024b229b6: Bootstrap complete.
I20240501 08:13:02.392491 3932 consensus_queue.cc:1035] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:13:02.392131 30670 raft_consensus.cc:2932] T 9820951bffc749139229e2b79c8ab597 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:13:02.392812 3909 ts_tablet_manager.cc:1367] T 32a54b70dcc447828ec293a9ff0afb80 P 31811708c6774037b82df44024b229b6: Time spent bootstrapping tablet: real 0.305s user 0.048s sys 0.016s
I20240501 08:13:02.393644 3909 raft_consensus.cc:348] T 32a54b70dcc447828ec293a9ff0afb80 P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
I20240501 08:13:02.393899 3909 raft_consensus.cc:729] T 32a54b70dcc447828ec293a9ff0afb80 P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31811708c6774037b82df44024b229b6, State: Initialized, Role: LEARNER
I20240501 08:13:02.394162 3909 consensus_queue.cc:260] T 32a54b70dcc447828ec293a9ff0afb80 P 31811708c6774037b82df44024b229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: NON_VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: true } }
I20240501 08:13:02.397538 3909 ts_tablet_manager.cc:1398] T 32a54b70dcc447828ec293a9ff0afb80 P 31811708c6774037b82df44024b229b6: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 08:13:02.399431 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6c04880ea87d4894bd3a63d94f4a69f4 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:13:02.399730 27781 tablet_copy_service.cc:342] P 6331f77dae9d452184e8cd0cbf5f1b43: Request end of tablet copy session 31811708c6774037b82df44024b229b6-32a54b70dcc447828ec293a9ff0afb80 received from {username='slave'} at 127.21.84.133:35347
I20240501 08:13:02.399840 27781 tablet_copy_service.cc:433] P 6331f77dae9d452184e8cd0cbf5f1b43: ending tablet copy session 31811708c6774037b82df44024b229b6-32a54b70dcc447828ec293a9ff0afb80 on tablet 32a54b70dcc447828ec293a9ff0afb80 with peer 31811708c6774037b82df44024b229b6
I20240501 08:13:02.399984 30668 raft_consensus.cc:2932] T bc5778711a4c479881088af790f5b7d5 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:13:02.407899 27615 catalog_manager.cc:5484] T 6c04880ea87d4894bd3a63d94f4a69f4 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 8 to 9, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:13:02.418386 27741 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c04880ea87d4894bd3a63d94f4a69f4 with delete_type TABLET_DATA_TOMBSTONED (TS 6331f77dae9d452184e8cd0cbf5f1b43 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:60422
I20240501 08:13:02.418543 30534 raft_consensus.cc:2932] T 6c04880ea87d4894bd3a63d94f4a69f4 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:13:02.419118 3946 tablet_replica.cc:330] T 6c04880ea87d4894bd3a63d94f4a69f4 P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:13:02.419394 3946 raft_consensus.cc:2229] T 6c04880ea87d4894bd3a63d94f4a69f4 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:02.419814 3946 raft_consensus.cc:2258] T 6c04880ea87d4894bd3a63d94f4a69f4 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:02.421506 3946 ts_tablet_manager.cc:1875] T 6c04880ea87d4894bd3a63d94f4a69f4 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:02.421993 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet 802538381de843ab8d326014a0b4ec15 (table rebalance_test_table_9 [id=e6efea2767a741beae5c93339e7712ba]) successfully deleted
I20240501 08:13:02.428917 28149 consensus_queue.cc:237] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:13:02.432900 3946 ts_tablet_manager.cc:1888] T 6c04880ea87d4894bd3a63d94f4a69f4 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 08:13:02.433125 3946 log.cc:1196] T 6c04880ea87d4894bd3a63d94f4a69f4 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/6c04880ea87d4894bd3a63d94f4a69f4
I20240501 08:13:02.433902 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet 6c04880ea87d4894bd3a63d94f4a69f4 (table rebalance_test_table_7 [id=1ae463b433bd4ed6bb9ab1bdc10dfdf6]) successfully deleted
I20240501 08:13:02.434414 28131 tablet_service.cc:1514] Processing DeleteTablet for tablet e6fb30f10c754d3a8e73e124ec59768b with delete_type TABLET_DATA_TOMBSTONED (TS 91d11eae5ef6432ca86eb8ed446b6ea6 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:55038
I20240501 08:13:02.434254 27615 catalog_manager.cc:5484] T bc5778711a4c479881088af790f5b7d5 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 7 to 8, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:13:02.435531 30802 raft_consensus.cc:1261] T 4b3e3d39f82549079293d98357884bf0 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:13:02.436049 3936 consensus_queue.cc:1035] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:13:02.439479 3475 raft_consensus.cc:2932] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:13:02.439764 30802 raft_consensus.cc:2932] T 4b3e3d39f82549079293d98357884bf0 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:13:02.441218 30537 raft_consensus.cc:2932] T e6fb30f10c754d3a8e73e124ec59768b P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:13:02.464659 27619 catalog_manager.cc:5484] T 4b3e3d39f82549079293d98357884bf0 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index 8 to 9, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:13:02.469925 28000 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b3e3d39f82549079293d98357884bf0 with delete_type TABLET_DATA_TOMBSTONED (TS aad81047009a4ffcbc450b60030583b5 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:41320
I20240501 08:13:02.474910 30934 raft_consensus.cc:1261] T 4b3e3d39f82549079293d98357884bf0 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:13:02.480062 28152 raft_consensus.cc:2932] T e6fb30f10c754d3a8e73e124ec59768b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:13:02.481093 3712 consensus_queue.cc:1035] T 7f7211c8c0e041bf89930aa50a4438ea P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:13:02.483703 27604 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4b3e3d39f82549079293d98357884bf0 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:13:02.495966 3475 consensus_queue.cc:1035] T 4b3e3d39f82549079293d98357884bf0 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:13:02.499161 30934 raft_consensus.cc:2932] T 4b3e3d39f82549079293d98357884bf0 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:13:02.507508 28149 consensus_queue.cc:237] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:13:02.515802 30672 raft_consensus.cc:1261] T bc5778711a4c479881088af790f5b7d5 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:13:02.516674 3475 consensus_queue.cc:1035] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:13:02.517866 30934 raft_consensus.cc:2932] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:13:02.520484 30537 raft_consensus.cc:1261] T bc5778711a4c479881088af790f5b7d5 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer 91d11eae5ef6432ca86eb8ed446b6ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:13:02.520712 3475 raft_consensus.cc:2932] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:13:02.521137 3928 consensus_queue.cc:1035] T bc5778711a4c479881088af790f5b7d5 P 91d11eae5ef6432ca86eb8ed446b6ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:13:02.522810 30535 raft_consensus.cc:2932] T bc5778711a4c479881088af790f5b7d5 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:13:02.525125 3712 consensus_queue.cc:1035] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:13:02.521953 30672 raft_consensus.cc:2932] T bc5778711a4c479881088af790f5b7d5 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:13:02.529415 27616 catalog_manager.cc:5031] ChangeConfig:REMOVE_PEER RPC for tablet 7f7211c8c0e041bf89930aa50a4438ea with cas_config_opid_index 8: aborting the task: latest config opid_index 9; task opid_index 8
I20240501 08:13:02.533242 3954 tablet_replica.cc:330] T e6fb30f10c754d3a8e73e124ec59768b P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:13:02.533529 3954 raft_consensus.cc:2229] T e6fb30f10c754d3a8e73e124ec59768b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:02.533797 3954 raft_consensus.cc:2258] T e6fb30f10c754d3a8e73e124ec59768b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:02.534194 28016 raft_consensus.cc:2932] T bc5778711a4c479881088af790f5b7d5 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:13:02.535334 3954 ts_tablet_manager.cc:1875] T e6fb30f10c754d3a8e73e124ec59768b P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:02.539253 3959 tablet_replica.cc:330] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:13:02.540809 27619 catalog_manager.cc:5484] T bc5778711a4c479881088af790f5b7d5 P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index 8 to 9, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New cstate: current_term: 1 leader_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } }
I20240501 08:13:02.548851 3236 raft_consensus.cc:1050] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11: attempting to promote NON_VOTER 6331f77dae9d452184e8cd0cbf5f1b43 to VOTER
I20240501 08:13:02.549753 3236 consensus_queue.cc:237] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:13:02.559022 27604 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bc5778711a4c479881088af790f5b7d5 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:13:02.561525 27871 tablet_service.cc:1514] Processing DeleteTablet for tablet b12559fc416a4f3cbffa018611226bcc with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 8 (current committed config index is 9)) from {username='slave'} at 127.0.0.1:48054
I20240501 08:13:02.563112 3954 ts_tablet_manager.cc:1888] T e6fb30f10c754d3a8e73e124ec59768b P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 08:13:02.563258 3954 log.cc:1196] T e6fb30f10c754d3a8e73e124ec59768b P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/e6fb30f10c754d3a8e73e124ec59768b
I20240501 08:13:02.564049 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet e6fb30f10c754d3a8e73e124ec59768b (table rebalance_test_table_19 [id=4fc7c32c4ee342e5804c1a70b815d952]) successfully deleted
I20240501 08:13:02.568266 27761 raft_consensus.cc:1261] T b6b0bcb4d07e4125bccf1711a094576e P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEARNER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:02.568883 3722 consensus_queue.cc:1035] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:02.570288 30672 raft_consensus.cc:1261] T b6b0bcb4d07e4125bccf1711a094576e P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:02.545420 28002 tablet_service.cc:1514] Processing DeleteTablet for tablet bc5778711a4c479881088af790f5b7d5 with delete_type TABLET_DATA_TOMBSTONED (TS aad81047009a4ffcbc450b60030583b5 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:41320
I20240501 08:13:02.570761 3959 raft_consensus.cc:2229] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:02.570986 3751 consensus_queue.cc:1035] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:02.571146 3959 raft_consensus.cc:2258] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:02.558537 28020 raft_consensus.cc:2932] T 9820951bffc749139229e2b79c8ab597 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:13:02.571213 30535 raft_consensus.cc:1261] T b6b0bcb4d07e4125bccf1711a094576e P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:02.571751 3751 consensus_queue.cc:1035] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:02.601284 3959 ts_tablet_manager.cc:1875] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:02.601830 3236 raft_consensus.cc:2932] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:13:02.602775 27607 catalog_manager.cc:4830] TS f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): tablet b12559fc416a4f3cbffa018611226bcc (table rebalance_test_table_17 [id=1121d4c7d5444060ac6df36744663e41]) successfully deleted
I20240501 08:13:02.604436 30672 raft_consensus.cc:2932] T b6b0bcb4d07e4125bccf1711a094576e P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:13:02.605700 27892 consensus_queue.cc:237] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:13:02.608054 30670 raft_consensus.cc:1261] T 9820951bffc749139229e2b79c8ab597 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:13:02.608289 30802 raft_consensus.cc:1261] T 9820951bffc749139229e2b79c8ab597 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:13:02.608757 3914 consensus_queue.cc:1035] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:13:02.609194 3914 consensus_queue.cc:1035] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:13:02.614974 27612 catalog_manager.cc:5484] T b6b0bcb4d07e4125bccf1711a094576e P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:13:02.616218 3712 raft_consensus.cc:2932] T 9820951bffc749139229e2b79c8ab597 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:13:02.619606 30668 raft_consensus.cc:2932] T 9820951bffc749139229e2b79c8ab597 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:13:02.623198 30802 raft_consensus.cc:2932] T 9820951bffc749139229e2b79c8ab597 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:13:02.625326 30934 consensus_queue.cc:237] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:13:02.627272 30535 raft_consensus.cc:2932] T b6b0bcb4d07e4125bccf1711a094576e P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:13:02.627334 27761 raft_consensus.cc:2932] T b6b0bcb4d07e4125bccf1711a094576e P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:13:02.635345 27619 catalog_manager.cc:5484] T 9820951bffc749139229e2b79c8ab597 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: config changed from index 8 to 9, VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) evicted. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } }
I20240501 08:13:02.635990 27607 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9820951bffc749139229e2b79c8ab597 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:13:02.641235 28002 tablet_service.cc:1514] Processing DeleteTablet for tablet 9820951bffc749139229e2b79c8ab597 with delete_type TABLET_DATA_TOMBSTONED (TS aad81047009a4ffcbc450b60030583b5 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:41320
I20240501 08:13:02.642714 30668 raft_consensus.cc:1261] T b6b0bcb4d07e4125bccf1711a094576e P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:13:02.643476 3722 consensus_queue.cc:1035] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:02.644158 3723 raft_consensus.cc:1050] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5: attempting to promote NON_VOTER a29cd590999b4f32849ff5689cd53c11 to VOTER
I20240501 08:13:02.644949 3723 consensus_queue.cc:237] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:13:02.649269 3722 raft_consensus.cc:2932] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:13:02.653720 30672 raft_consensus.cc:2932] T b6b0bcb4d07e4125bccf1711a094576e P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:13:02.654577 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b6b0bcb4d07e4125bccf1711a094576e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:13:02.663216 27758 raft_consensus.cc:1261] T b6b0bcb4d07e4125bccf1711a094576e P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer a29cd590999b4f32849ff5689cd53c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:13:02.664045 3963 consensus_queue.cc:1035] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:02.666137 27758 raft_consensus.cc:2932] T b6b0bcb4d07e4125bccf1711a094576e P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } }
I20240501 08:13:02.667790 28149 raft_consensus.cc:1261] T 5eccb9416d7d438893307f13dd4e73dc P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:13:02.668478 3723 consensus_queue.cc:1035] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:13:02.670629 30802 raft_consensus.cc:1261] T 5eccb9416d7d438893307f13dd4e73dc P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:13:02.671183 3816 consensus_queue.cc:1035] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:13:02.673023 27612 catalog_manager.cc:5484] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index 3 to 4, VOTER 31811708c6774037b82df44024b229b6 (127.21.84.133) evicted. New cstate: current_term: 1 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:02.675436 30934 raft_consensus.cc:1261] T 5eccb9416d7d438893307f13dd4e73dc P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:13:02.676128 3816 consensus_queue.cc:1035] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:13:02.678757 3959 ts_tablet_manager.cc:1888] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 08:13:02.683776 3959 log.cc:1196] T 4b3e3d39f82549079293d98357884bf0 P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/4b3e3d39f82549079293d98357884bf0
I20240501 08:13:02.684939 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet 4b3e3d39f82549079293d98357884bf0 (table rebalance_test_table_3 [id=56fa27c3d3de4793aadbcaf54cdb0ac5]) successfully deleted
I20240501 08:13:02.685623 3959 tablet_replica.cc:330] T bc5778711a4c479881088af790f5b7d5 P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:13:02.686600 3959 raft_consensus.cc:2229] T bc5778711a4c479881088af790f5b7d5 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:02.687269 3959 raft_consensus.cc:2258] T bc5778711a4c479881088af790f5b7d5 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:02.688733 3959 ts_tablet_manager.cc:1875] T bc5778711a4c479881088af790f5b7d5 P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:02.691293 30514 tablet_service.cc:1514] Processing DeleteTablet for tablet b6b0bcb4d07e4125bccf1711a094576e with delete_type TABLET_DATA_TOMBSTONED (TS 31811708c6774037b82df44024b229b6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45078
I20240501 08:13:02.704751 3977 tablet_replica.cc:330] T b6b0bcb4d07e4125bccf1711a094576e P 31811708c6774037b82df44024b229b6: stopping tablet replica
I20240501 08:13:02.705009 3977 raft_consensus.cc:2229] T b6b0bcb4d07e4125bccf1711a094576e P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:02.705286 3977 raft_consensus.cc:2258] T b6b0bcb4d07e4125bccf1711a094576e P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:02.714273 3977 ts_tablet_manager.cc:1875] T b6b0bcb4d07e4125bccf1711a094576e P 31811708c6774037b82df44024b229b6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:02.736017 3977 ts_tablet_manager.cc:1888] T b6b0bcb4d07e4125bccf1711a094576e P 31811708c6774037b82df44024b229b6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:13:02.736326 3712 raft_consensus.cc:1050] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af: attempting to promote NON_VOTER a29cd590999b4f32849ff5689cd53c11 to VOTER
I20240501 08:13:02.736555 3977 log.cc:1196] T b6b0bcb4d07e4125bccf1711a094576e P 31811708c6774037b82df44024b229b6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/wals/b6b0bcb4d07e4125bccf1711a094576e
I20240501 08:13:02.739298 27607 catalog_manager.cc:4830] TS 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): tablet b6b0bcb4d07e4125bccf1711a094576e (table rebalancer_extra_table_18 [id=c906ec3019644e00bd7c63c8bcc3d323]) successfully deleted
I20240501 08:13:02.741082 3712 consensus_queue.cc:237] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:13:02.745074 3959 ts_tablet_manager.cc:1888] T bc5778711a4c479881088af790f5b7d5 P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 08:13:02.748281 30801 raft_consensus.cc:1261] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:13:02.748872 3914 consensus_queue.cc:1035] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:13:02.749650 3959 log.cc:1196] T bc5778711a4c479881088af790f5b7d5 P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/bc5778711a4c479881088af790f5b7d5
I20240501 08:13:02.750633 3959 tablet_replica.cc:330] T 9820951bffc749139229e2b79c8ab597 P aad81047009a4ffcbc450b60030583b5: stopping tablet replica
I20240501 08:13:02.751507 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet bc5778711a4c479881088af790f5b7d5 (table rebalance_test_table_5 [id=b19758166c02438c9fc4cfa835acc028]) successfully deleted
I20240501 08:13:02.752617 30934 raft_consensus.cc:1261] T 0e06cf862c954e8fb1ca087ec5184497 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEARNER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:13:02.753052 3959 raft_consensus.cc:2229] T 9820951bffc749139229e2b79c8ab597 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:02.759027 3959 raft_consensus.cc:2258] T 9820951bffc749139229e2b79c8ab597 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:02.762208 3959 ts_tablet_manager.cc:1875] T 9820951bffc749139229e2b79c8ab597 P aad81047009a4ffcbc450b60030583b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:02.765619 3914 consensus_queue.cc:1035] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:13:02.762724 28020 raft_consensus.cc:1261] T 0e06cf862c954e8fb1ca087ec5184497 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:13:02.768919 3914 consensus_queue.cc:1035] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:13:02.782037 3723 raft_consensus.cc:2932] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:13:02.789209 30802 raft_consensus.cc:2932] T 5eccb9416d7d438893307f13dd4e73dc P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:13:02.790048 28149 raft_consensus.cc:2932] T 5eccb9416d7d438893307f13dd4e73dc P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:13:02.791031 27616 catalog_manager.cc:5484] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 7 to 8, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:02.799508 3959 ts_tablet_manager.cc:1888] T 9820951bffc749139229e2b79c8ab597 P aad81047009a4ffcbc450b60030583b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 08:13:02.801118 3959 log.cc:1196] T 9820951bffc749139229e2b79c8ab597 P aad81047009a4ffcbc450b60030583b5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/wals/9820951bffc749139229e2b79c8ab597
I20240501 08:13:02.802311 27606 catalog_manager.cc:4830] TS aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): tablet 9820951bffc749139229e2b79c8ab597 (table rebalance_test_table_13 [id=86d1b715252040a3bc0f714b3db3f4ff]) successfully deleted
I20240501 08:13:02.802574 30533 raft_consensus.cc:1203] T 32a54b70dcc447828ec293a9ff0afb80 P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 08:13:02.806529 30936 raft_consensus.cc:2932] T 5eccb9416d7d438893307f13dd4e73dc P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:13:02.809887 28016 consensus_queue.cc:237] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:13:02.813390 30802 raft_consensus.cc:1261] T 5eccb9416d7d438893307f13dd4e73dc P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:13:02.814337 3543 consensus_queue.cc:1035] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:13:02.818514 3688 raft_consensus.cc:2932] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:13:02.823143 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5eccb9416d7d438893307f13dd4e73dc with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:13:02.823091 3712 raft_consensus.cc:2932] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:13:02.823801 30801 raft_consensus.cc:2932] T 5eccb9416d7d438893307f13dd4e73dc P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:13:02.825834 30934 raft_consensus.cc:1261] T 5eccb9416d7d438893307f13dd4e73dc P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 08:13:02.826392 3688 consensus_queue.cc:1035] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:13:02.826582 27616 catalog_manager.cc:5484] T 5eccb9416d7d438893307f13dd4e73dc P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 8 to 9, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:02.829861 30937 raft_consensus.cc:2932] T 5eccb9416d7d438893307f13dd4e73dc P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:13:02.836763 30801 raft_consensus.cc:2932] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:13:02.840803 28131 tablet_service.cc:1514] Processing DeleteTablet for tablet 5eccb9416d7d438893307f13dd4e73dc with delete_type TABLET_DATA_TOMBSTONED (TS 91d11eae5ef6432ca86eb8ed446b6ea6 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:55038
I20240501 08:13:02.842062 3954 tablet_replica.cc:330] T 5eccb9416d7d438893307f13dd4e73dc P 91d11eae5ef6432ca86eb8ed446b6ea6: stopping tablet replica
I20240501 08:13:02.842283 3954 raft_consensus.cc:2229] T 5eccb9416d7d438893307f13dd4e73dc P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:02.842480 3954 raft_consensus.cc:2258] T 5eccb9416d7d438893307f13dd4e73dc P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:02.845273 3954 ts_tablet_manager.cc:1875] T 5eccb9416d7d438893307f13dd4e73dc P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:02.852051 3954 ts_tablet_manager.cc:1888] T 5eccb9416d7d438893307f13dd4e73dc P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 08:13:02.852272 3954 log.cc:1196] T 5eccb9416d7d438893307f13dd4e73dc P 91d11eae5ef6432ca86eb8ed446b6ea6: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/wals/5eccb9416d7d438893307f13dd4e73dc
I20240501 08:13:02.854374 27604 catalog_manager.cc:4830] TS 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): tablet 5eccb9416d7d438893307f13dd4e73dc (table rebalance_test_table_11 [id=05c446d8d19b4077b3441b0349357c6e]) successfully deleted
I20240501 08:13:02.853984 27616 catalog_manager.cc:5484] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: config changed from index 5 to 6, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:13:02.857975 30934 raft_consensus.cc:2932] T 0e06cf862c954e8fb1ca087ec5184497 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:13:02.865043 28020 raft_consensus.cc:2932] T 0e06cf862c954e8fb1ca087ec5184497 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:13:02.872177 27889 consensus_queue.cc:237] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:13:02.878772 28016 raft_consensus.cc:1261] T 0e06cf862c954e8fb1ca087ec5184497 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:13:02.879561 3957 consensus_queue.cc:1035] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:13:02.886425 3914 raft_consensus.cc:2932] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:13:02.889282 30937 raft_consensus.cc:1261] T 0e06cf862c954e8fb1ca087ec5184497 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer f1525cbc82a24d13b65f136695c903af: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:13:02.892922 27607 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0e06cf862c954e8fb1ca087ec5184497 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:13:02.896239 28016 raft_consensus.cc:2932] T 0e06cf862c954e8fb1ca087ec5184497 P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:13:02.907989 27615 catalog_manager.cc:5484] T 0e06cf862c954e8fb1ca087ec5184497 P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 6 to 7, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New cstate: current_term: 1 leader_uuid: "f1525cbc82a24d13b65f136695c903af" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:13:02.909817 3914 consensus_queue.cc:1035] T 0e06cf862c954e8fb1ca087ec5184497 P f1525cbc82a24d13b65f136695c903af [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:13:02.915861 30934 raft_consensus.cc:2932] T 0e06cf862c954e8fb1ca087ec5184497 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 64998323a1f4482d8704bf20594d13a5 (127.21.84.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } }
I20240501 08:13:02.917797 30784 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e06cf862c954e8fb1ca087ec5184497 with delete_type TABLET_DATA_TOMBSTONED (TS 64998323a1f4482d8704bf20594d13a5 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38952
I20240501 08:13:02.937388 3987 tablet_replica.cc:330] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5: stopping tablet replica
I20240501 08:13:02.937731 3987 raft_consensus.cc:2229] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:02.938290 3987 raft_consensus.cc:2258] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:02.939173 3987 ts_tablet_manager.cc:1875] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:02.949858 3987 ts_tablet_manager.cc:1888] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 08:13:02.955490 3987 log.cc:1196] T 0e06cf862c954e8fb1ca087ec5184497 P 64998323a1f4482d8704bf20594d13a5: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal/wals/0e06cf862c954e8fb1ca087ec5184497
I20240501 08:13:02.956630 27605 catalog_manager.cc:4830] TS 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): tablet 0e06cf862c954e8fb1ca087ec5184497 (table rebalancer_extra_table_26 [id=444d193c9c4c4c16969e6c88e6e590c9]) successfully deleted
I20240501 08:13:03.111254 30533 consensus_queue.cc:237] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:13:03.116291 27889 raft_consensus.cc:1261] T 7affd631d1e444eaaf8a4c51267a204f P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:13:03.116873 4034 consensus_queue.cc:1035] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:13:03.120802 3952 raft_consensus.cc:2932] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:13:03.123622 30934 raft_consensus.cc:1261] T 7affd631d1e444eaaf8a4c51267a204f P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:13:03.124931 27889 raft_consensus.cc:2932] T 7affd631d1e444eaaf8a4c51267a204f P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
W20240501 08:13:03.126991 30472 consensus_peers.cc:485] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 -> Peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Couldn't send request to peer aad81047009a4ffcbc450b60030583b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7affd631d1e444eaaf8a4c51267a204f. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:03.129506 3952 consensus_queue.cc:1035] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20240501 08:13:03.135655 27616 catalog_manager.cc:5484] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index 4 to 5, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:03.142072 30934 raft_consensus.cc:2932] T 7affd631d1e444eaaf8a4c51267a204f P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aad81047009a4ffcbc450b60030583b5 (127.21.84.131) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } } }
I20240501 08:13:03.161716 28016 consensus_queue.cc:237] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
W20240501 08:13:03.164302 27955 consensus_peers.cc:485] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 -> Peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Couldn't send request to peer 91d11eae5ef6432ca86eb8ed446b6ea6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 95cb5060152e4237988a4c5723a62601. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:03.164273 27758 raft_consensus.cc:1261] T 95cb5060152e4237988a4c5723a62601 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:13:03.164388 30934 raft_consensus.cc:1261] T 95cb5060152e4237988a4c5723a62601 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:13:03.164865 3920 consensus_queue.cc:1035] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:13:03.165163 3688 consensus_queue.cc:1035] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:13:03.167903 3688 raft_consensus.cc:2932] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:13:03.170949 30934 raft_consensus.cc:2932] T 95cb5060152e4237988a4c5723a62601 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:13:03.171097 27758 raft_consensus.cc:2932] T 95cb5060152e4237988a4c5723a62601 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } } }
I20240501 08:13:03.175511 27612 catalog_manager.cc:5484] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 4 to 5, NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) added. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:03.267230 30934 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f63215605ade41a5bc8d97a35ce33a31"
dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35132
I20240501 08:13:03.267439 30934 raft_consensus.cc:595] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:13:03.280275 3931 raft_consensus.cc:1050] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43: attempting to promote NON_VOTER 31811708c6774037b82df44024b229b6 to VOTER
I20240501 08:13:03.281147 3931 consensus_queue.cc:237] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } }
I20240501 08:13:03.284799 30801 raft_consensus.cc:1261] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:13:03.285580 3246 consensus_queue.cc:1035] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:13:03.286959 27892 raft_consensus.cc:1261] T 32a54b70dcc447828ec293a9ff0afb80 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:13:03.287633 30535 raft_consensus.cc:1261] T 32a54b70dcc447828ec293a9ff0afb80 P 31811708c6774037b82df44024b229b6 [term 1 LEARNER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 08:13:03.289290 3654 consensus_queue.cc:1035] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:13:03.289798 3654 consensus_queue.cc:1035] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:13:03.294813 3722 raft_consensus.cc:979] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11: : Instructing follower 6331f77dae9d452184e8cd0cbf5f1b43 to start an election
I20240501 08:13:03.294957 3722 raft_consensus.cc:1067] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Signalling peer 6331f77dae9d452184e8cd0cbf5f1b43 to start an election
I20240501 08:13:03.295595 27758 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f63215605ade41a5bc8d97a35ce33a31"
dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
from {username='slave'} at 127.21.84.136:34027
I20240501 08:13:03.295790 27758 raft_consensus.cc:482] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:03.295884 27758 raft_consensus.cc:3037] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:03.298763 27758 raft_consensus.cc:504] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:13:03.300019 30934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63215605ade41a5bc8d97a35ce33a31" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:13:03.300222 30934 raft_consensus.cc:3032] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:03.300290 30934 raft_consensus.cc:729] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:13:03.300529 30934 consensus_queue.cc:260] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:13:03.300902 30934 raft_consensus.cc:3037] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:03.301400 3655 raft_consensus.cc:2932] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:13:03.304219 28149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63215605ade41a5bc8d97a35ce33a31" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:13:03.304433 28149 raft_consensus.cc:3037] T f63215605ade41a5bc8d97a35ce33a31 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:03.305356 30934 raft_consensus.cc:2454] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 2.
I20240501 08:13:03.307046 28149 raft_consensus.cc:2454] T f63215605ade41a5bc8d97a35ce33a31 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 2.
I20240501 08:13:03.307678 30533 raft_consensus.cc:2932] T 32a54b70dcc447828ec293a9ff0afb80 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:13:03.308662 27696 leader_election.cc:302] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6331f77dae9d452184e8cd0cbf5f1b43, 91d11eae5ef6432ca86eb8ed446b6ea6, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:13:03.310127 30534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63215605ade41a5bc8d97a35ce33a31" candidate_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:13:03.310345 30534 raft_consensus.cc:3037] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:03.312693 30801 raft_consensus.cc:2932] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:13:03.313668 3931 raft_consensus.cc:2781] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:03.313894 30534 raft_consensus.cc:2454] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6331f77dae9d452184e8cd0cbf5f1b43 in term 2.
I20240501 08:13:03.322790 27612 catalog_manager.cc:5484] T 32a54b70dcc447828ec293a9ff0afb80 P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index 7 to 8, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:13:03.340211 27758 leader_election.cc:288] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43 [CANDIDATE]: Term 2 election: Requested vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:13:03.358892 3931 raft_consensus.cc:686] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 LEADER]: Becoming Leader. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:13:03.359321 3931 consensus_queue.cc:237] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:13:03.370946 27616 catalog_manager.cc:5484] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: term changed from 1 to 2, leader changed from a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) to 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129). New cstate: current_term: 2 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:03.372411 27892 raft_consensus.cc:2932] T 32a54b70dcc447828ec293a9ff0afb80 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 31811708c6774037b82df44024b229b6 (127.21.84.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:13:03.553043 27761 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "32a54b70dcc447828ec293a9ff0afb80"
dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36990
I20240501 08:13:03.553280 27761 raft_consensus.cc:595] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:13:03.693100 3931 raft_consensus.cc:979] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43: : Instructing follower 64998323a1f4482d8704bf20594d13a5 to start an election
I20240501 08:13:03.693277 4099 raft_consensus.cc:1067] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Signalling peer 64998323a1f4482d8704bf20594d13a5 to start an election
I20240501 08:13:03.693917 30801 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "32a54b70dcc447828ec293a9ff0afb80"
dest_uuid: "64998323a1f4482d8704bf20594d13a5"
from {username='slave'} at 127.21.84.129:58333
I20240501 08:13:03.694181 30801 raft_consensus.cc:482] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:03.694293 30801 raft_consensus.cc:3037] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:03.696257 30801 raft_consensus.cc:504] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } }
I20240501 08:13:03.697194 30801 leader_election.cc:288] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 election: Requested vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:13:03.697692 30534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a54b70dcc447828ec293a9ff0afb80" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:13:03.697953 30534 raft_consensus.cc:3037] T 32a54b70dcc447828ec293a9ff0afb80 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:03.700867 30534 raft_consensus.cc:2454] T 32a54b70dcc447828ec293a9ff0afb80 P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 2.
I20240501 08:13:03.701433 27892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a54b70dcc447828ec293a9ff0afb80" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "f1525cbc82a24d13b65f136695c903af"
I20240501 08:13:03.701792 27892 raft_consensus.cc:3037] T 32a54b70dcc447828ec293a9ff0afb80 P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:03.705858 27892 raft_consensus.cc:2454] T 32a54b70dcc447828ec293a9ff0afb80 P f1525cbc82a24d13b65f136695c903af [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 2.
I20240501 08:13:03.706673 30740 leader_election.cc:302] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 64998323a1f4482d8704bf20594d13a5, f1525cbc82a24d13b65f136695c903af; no voters:
I20240501 08:13:03.707196 3884 raft_consensus.cc:2781] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:03.707376 3884 raft_consensus.cc:686] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5 [term 2 LEADER]: Becoming Leader. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Running, Role: LEADER
I20240501 08:13:03.707371 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a54b70dcc447828ec293a9ff0afb80" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "6331f77dae9d452184e8cd0cbf5f1b43"
I20240501 08:13:03.707598 27761 raft_consensus.cc:3032] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:03.707656 27761 raft_consensus.cc:729] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6331f77dae9d452184e8cd0cbf5f1b43, State: Running, Role: LEADER
I20240501 08:13:03.707939 4126 ts_tablet_manager.cc:902] T 7affd631d1e444eaaf8a4c51267a204f P aad81047009a4ffcbc450b60030583b5: Initiating tablet copy from peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:13:03.707762 3884 consensus_queue.cc:237] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } }
I20240501 08:13:03.708402 27761 consensus_queue.cc:260] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } }
I20240501 08:13:03.708951 27761 raft_consensus.cc:3037] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:03.715340 27761 raft_consensus.cc:2454] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 2.
I20240501 08:13:03.718405 27612 catalog_manager.cc:5484] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: term changed from 1 to 2, leader changed from 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) to 64998323a1f4482d8704bf20594d13a5 (127.21.84.135). New cstate: current_term: 2 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:03.754266 4126 tablet_copy_client.cc:308] T 7affd631d1e444eaaf8a4c51267a204f P aad81047009a4ffcbc450b60030583b5: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.133:38293
I20240501 08:13:03.755373 30559 tablet_copy_service.cc:140] P 31811708c6774037b82df44024b229b6: Received BeginTabletCopySession request for tablet 7affd631d1e444eaaf8a4c51267a204f from peer aad81047009a4ffcbc450b60030583b5 ({username='slave'} at 127.21.84.131:56895)
I20240501 08:13:03.755544 30559 tablet_copy_service.cc:161] P 31811708c6774037b82df44024b229b6: Beginning new tablet copy session on tablet 7affd631d1e444eaaf8a4c51267a204f from peer aad81047009a4ffcbc450b60030583b5 at {username='slave'} at 127.21.84.131:56895: session id = aad81047009a4ffcbc450b60030583b5-7affd631d1e444eaaf8a4c51267a204f
I20240501 08:13:03.759493 30559 tablet_copy_source_session.cc:205] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:03.760875 4126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7affd631d1e444eaaf8a4c51267a204f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:03.767848 4126 tablet_copy_client.cc:783] T 7affd631d1e444eaaf8a4c51267a204f P aad81047009a4ffcbc450b60030583b5: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:03.768112 4126 tablet_copy_client.cc:647] T 7affd631d1e444eaaf8a4c51267a204f P aad81047009a4ffcbc450b60030583b5: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:03.770879 4126 tablet_copy_client.cc:523] T 7affd631d1e444eaaf8a4c51267a204f P aad81047009a4ffcbc450b60030583b5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:03.774331 4126 tablet_bootstrap.cc:492] T 7affd631d1e444eaaf8a4c51267a204f P aad81047009a4ffcbc450b60030583b5: Bootstrap starting.
I20240501 08:13:03.785182 4126 tablet_bootstrap.cc:492] T 7affd631d1e444eaaf8a4c51267a204f P aad81047009a4ffcbc450b60030583b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:03.785709 4126 tablet_bootstrap.cc:492] T 7affd631d1e444eaaf8a4c51267a204f P aad81047009a4ffcbc450b60030583b5: Bootstrap complete.
I20240501 08:13:03.786067 4126 ts_tablet_manager.cc:1367] T 7affd631d1e444eaaf8a4c51267a204f P aad81047009a4ffcbc450b60030583b5: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20240501 08:13:03.786800 4126 raft_consensus.cc:348] T 7affd631d1e444eaaf8a4c51267a204f P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:13:03.787084 4126 raft_consensus.cc:729] T 7affd631d1e444eaaf8a4c51267a204f P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aad81047009a4ffcbc450b60030583b5, State: Initialized, Role: LEARNER
I20240501 08:13:03.787302 4126 consensus_queue.cc:260] T 7affd631d1e444eaaf8a4c51267a204f P aad81047009a4ffcbc450b60030583b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: NON_VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: true } }
I20240501 08:13:03.788072 4126 ts_tablet_manager.cc:1398] T 7affd631d1e444eaaf8a4c51267a204f P aad81047009a4ffcbc450b60030583b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:13:03.789589 30559 tablet_copy_service.cc:342] P 31811708c6774037b82df44024b229b6: Request end of tablet copy session aad81047009a4ffcbc450b60030583b5-7affd631d1e444eaaf8a4c51267a204f received from {username='slave'} at 127.21.84.131:56895
I20240501 08:13:03.789710 30559 tablet_copy_service.cc:433] P 31811708c6774037b82df44024b229b6: ending tablet copy session aad81047009a4ffcbc450b60030583b5-7affd631d1e444eaaf8a4c51267a204f on tablet 7affd631d1e444eaaf8a4c51267a204f with peer aad81047009a4ffcbc450b60030583b5
I20240501 08:13:03.799008 4138 ts_tablet_manager.cc:902] T 95cb5060152e4237988a4c5723a62601 P 91d11eae5ef6432ca86eb8ed446b6ea6: Initiating tablet copy from peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:13:03.804019 4138 tablet_copy_client.cc:308] T 95cb5060152e4237988a4c5723a62601 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Beginning tablet copy session from remote peer at address 127.21.84.131:36305
I20240501 08:13:03.804800 28042 tablet_copy_service.cc:140] P aad81047009a4ffcbc450b60030583b5: Received BeginTabletCopySession request for tablet 95cb5060152e4237988a4c5723a62601 from peer 91d11eae5ef6432ca86eb8ed446b6ea6 ({username='slave'} at 127.21.84.132:57283)
I20240501 08:13:03.804963 28042 tablet_copy_service.cc:161] P aad81047009a4ffcbc450b60030583b5: Beginning new tablet copy session on tablet 95cb5060152e4237988a4c5723a62601 from peer 91d11eae5ef6432ca86eb8ed446b6ea6 at {username='slave'} at 127.21.84.132:57283: session id = 91d11eae5ef6432ca86eb8ed446b6ea6-95cb5060152e4237988a4c5723a62601
I20240501 08:13:03.807572 28042 tablet_copy_source_session.cc:205] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:03.809020 4138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95cb5060152e4237988a4c5723a62601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:03.817865 4138 tablet_copy_client.cc:783] T 95cb5060152e4237988a4c5723a62601 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:03.818145 4138 tablet_copy_client.cc:647] T 95cb5060152e4237988a4c5723a62601 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:03.820145 4138 tablet_copy_client.cc:523] T 95cb5060152e4237988a4c5723a62601 P 91d11eae5ef6432ca86eb8ed446b6ea6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:03.823221 4138 tablet_bootstrap.cc:492] T 95cb5060152e4237988a4c5723a62601 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap starting.
I20240501 08:13:03.830986 4138 tablet_bootstrap.cc:492] T 95cb5060152e4237988a4c5723a62601 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:03.831511 4138 tablet_bootstrap.cc:492] T 95cb5060152e4237988a4c5723a62601 P 91d11eae5ef6432ca86eb8ed446b6ea6: Bootstrap complete.
I20240501 08:13:03.831822 4138 ts_tablet_manager.cc:1367] T 95cb5060152e4237988a4c5723a62601 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent bootstrapping tablet: real 0.009s user 0.009s sys 0.000s
I20240501 08:13:03.833679 4138 raft_consensus.cc:348] T 95cb5060152e4237988a4c5723a62601 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:13:03.833933 4138 raft_consensus.cc:729] T 95cb5060152e4237988a4c5723a62601 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 91d11eae5ef6432ca86eb8ed446b6ea6, State: Initialized, Role: LEARNER
I20240501 08:13:03.834154 4138 consensus_queue.cc:260] T 95cb5060152e4237988a4c5723a62601 P 91d11eae5ef6432ca86eb8ed446b6ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: NON_VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: true } }
I20240501 08:13:03.835086 4138 ts_tablet_manager.cc:1398] T 95cb5060152e4237988a4c5723a62601 P 91d11eae5ef6432ca86eb8ed446b6ea6: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:13:03.835939 28042 tablet_copy_service.cc:342] P aad81047009a4ffcbc450b60030583b5: Request end of tablet copy session 91d11eae5ef6432ca86eb8ed446b6ea6-95cb5060152e4237988a4c5723a62601 received from {username='slave'} at 127.21.84.132:57283
I20240501 08:13:03.836623 28042 tablet_copy_service.cc:433] P aad81047009a4ffcbc450b60030583b5: ending tablet copy session 91d11eae5ef6432ca86eb8ed446b6ea6-95cb5060152e4237988a4c5723a62601 on tablet 95cb5060152e4237988a4c5723a62601 with peer 91d11eae5ef6432ca86eb8ed446b6ea6
I20240501 08:13:03.862471 30533 raft_consensus.cc:1261] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:13:03.874866 4098 consensus_queue.cc:1035] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:13:03.901424 28149 raft_consensus.cc:1261] T f63215605ade41a5bc8d97a35ce33a31 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:13:03.902391 4098 consensus_queue.cc:1035] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:13:03.927014 30937 raft_consensus.cc:1261] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:13:03.928048 3931 consensus_queue.cc:1035] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:13:03.941682 27761 consensus_queue.cc:237] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:13:03.943354 28149 raft_consensus.cc:1261] T f63215605ade41a5bc8d97a35ce33a31 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:13:03.943406 30534 raft_consensus.cc:1261] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Refusing update from remote peer 6331f77dae9d452184e8cd0cbf5f1b43: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 08:13:03.943884 4098 consensus_queue.cc:1035] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:13:03.944125 4099 consensus_queue.cc:1035] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:13:03.946288 4098 raft_consensus.cc:2932] T f63215605ade41a5bc8d97a35ce33a31 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:13:03.946978 28149 raft_consensus.cc:2932] T f63215605ade41a5bc8d97a35ce33a31 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:13:03.948582 30534 raft_consensus.cc:2932] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:13:03.952406 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f63215605ade41a5bc8d97a35ce33a31 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:13:03.954456 27612 catalog_manager.cc:5484] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index 6 to 8, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New cstate: current_term: 2 leader_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:13:03.958150 30916 tablet_service.cc:1514] Processing DeleteTablet for tablet f63215605ade41a5bc8d97a35ce33a31 with delete_type TABLET_DATA_TOMBSTONED (TS a29cd590999b4f32849ff5689cd53c11 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:54036
I20240501 08:13:03.960078 4169 tablet_replica.cc:330] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11: stopping tablet replica
I20240501 08:13:03.960354 4169 raft_consensus.cc:2229] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:03.960574 4169 raft_consensus.cc:2258] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:03.961565 4169 ts_tablet_manager.cc:1875] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:03.966660 4169 ts_tablet_manager.cc:1888] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240501 08:13:03.966787 4169 log.cc:1196] T f63215605ade41a5bc8d97a35ce33a31 P a29cd590999b4f32849ff5689cd53c11: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/wals/f63215605ade41a5bc8d97a35ce33a31
I20240501 08:13:03.967489 27606 catalog_manager.cc:4830] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): tablet f63215605ade41a5bc8d97a35ce33a31 (table rebalancer_extra_table_44 [id=a727b21f159445ba8749f9bf1bec2b3d]) successfully deleted
I20240501 08:13:04.130386 27889 raft_consensus.cc:1261] T 32a54b70dcc447828ec293a9ff0afb80 P f1525cbc82a24d13b65f136695c903af [term 2 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:13:04.131002 4100 consensus_queue.cc:1035] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:13:04.218394 30533 raft_consensus.cc:1261] T 32a54b70dcc447828ec293a9ff0afb80 P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:13:04.218958 4100 consensus_queue.cc:1035] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:13:04.223608 28016 raft_consensus.cc:1203] T 7affd631d1e444eaaf8a4c51267a204f P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 08:13:04.226040 27761 raft_consensus.cc:1261] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 08:13:04.226658 4100 consensus_queue.cc:1035] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 08:13:04.228439 30802 consensus_queue.cc:237] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } }
I20240501 08:13:04.239974 27889 raft_consensus.cc:1261] T 32a54b70dcc447828ec293a9ff0afb80 P f1525cbc82a24d13b65f136695c903af [term 2 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 08:13:04.240252 30533 raft_consensus.cc:1261] T 32a54b70dcc447828ec293a9ff0afb80 P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 08:13:04.241122 4100 consensus_queue.cc:1035] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:13:04.242471 4100 consensus_queue.cc:1035] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:13:04.244019 4170 raft_consensus.cc:2932] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:13:04.244705 27889 raft_consensus.cc:2932] T 32a54b70dcc447828ec293a9ff0afb80 P f1525cbc82a24d13b65f136695c903af [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:13:04.245510 30533 raft_consensus.cc:2932] T 32a54b70dcc447828ec293a9ff0afb80 P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:13:04.250680 27605 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 32a54b70dcc447828ec293a9ff0afb80 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:13:04.253548 27615 catalog_manager.cc:5484] T 32a54b70dcc447828ec293a9ff0afb80 P f1525cbc82a24d13b65f136695c903af reported cstate change: config changed from index 8 to 10, VOTER 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) evicted. New cstate: current_term: 2 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } }
I20240501 08:13:04.258960 27741 tablet_service.cc:1514] Processing DeleteTablet for tablet 32a54b70dcc447828ec293a9ff0afb80 with delete_type TABLET_DATA_TOMBSTONED (TS 6331f77dae9d452184e8cd0cbf5f1b43 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:60422
I20240501 08:13:04.260653 4210 tablet_replica.cc:330] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43: stopping tablet replica
I20240501 08:13:04.265347 4210 raft_consensus.cc:2229] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:04.265637 4210 raft_consensus.cc:2258] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:04.265856 28149 raft_consensus.cc:1203] T 95cb5060152e4237988a4c5723a62601 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 08:13:04.268774 4210 ts_tablet_manager.cc:1875] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:04.276286 4210 ts_tablet_manager.cc:1888] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 08:13:04.276432 4210 log.cc:1196] T 32a54b70dcc447828ec293a9ff0afb80 P 6331f77dae9d452184e8cd0cbf5f1b43: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/wals/32a54b70dcc447828ec293a9ff0afb80
I20240501 08:13:04.277205 27606 catalog_manager.cc:4830] TS 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): tablet 32a54b70dcc447828ec293a9ff0afb80 (table rebalance_test_table_15 [id=d3eb6ca7d8cf4bfdb114edb23abb2ed2]) successfully deleted
I20240501 08:13:04.671655 4209 raft_consensus.cc:1050] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5: attempting to promote NON_VOTER 91d11eae5ef6432ca86eb8ed446b6ea6 to VOTER
I20240501 08:13:04.672398 4209 consensus_queue.cc:237] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:13:04.676532 28152 raft_consensus.cc:1261] T 95cb5060152e4237988a4c5723a62601 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 LEARNER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:13:04.676823 27761 raft_consensus.cc:1261] T 95cb5060152e4237988a4c5723a62601 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:13:04.677225 30934 raft_consensus.cc:1261] T 95cb5060152e4237988a4c5723a62601 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:13:04.677757 3723 consensus_queue.cc:1035] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:13:04.678045 4208 consensus_queue.cc:1035] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:13:04.678582 4209 consensus_queue.cc:1035] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:13:04.680788 4209 raft_consensus.cc:2932] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:13:04.681435 27761 raft_consensus.cc:2932] T 95cb5060152e4237988a4c5723a62601 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:13:04.682967 28152 raft_consensus.cc:2932] T 95cb5060152e4237988a4c5723a62601 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:13:04.685348 30934 raft_consensus.cc:2932] T 95cb5060152e4237988a4c5723a62601 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:13:04.689841 27615 catalog_manager.cc:5484] T 95cb5060152e4237988a4c5723a62601 P 6331f77dae9d452184e8cd0cbf5f1b43 reported cstate change: config changed from index 5 to 6, 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:13:04.696281 28020 consensus_queue.cc:237] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:13:04.698035 28149 raft_consensus.cc:1261] T 95cb5060152e4237988a4c5723a62601 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:13:04.702340 3723 consensus_queue.cc:1035] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:13:04.702577 27761 raft_consensus.cc:1261] T 95cb5060152e4237988a4c5723a62601 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Refusing update from remote peer aad81047009a4ffcbc450b60030583b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:13:04.704150 3723 consensus_queue.cc:1035] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:13:04.705075 3688 raft_consensus.cc:2932] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:13:04.706265 28152 raft_consensus.cc:2932] T 95cb5060152e4237988a4c5723a62601 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:13:04.711236 27606 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 95cb5060152e4237988a4c5723a62601 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:13:04.715817 27615 catalog_manager.cc:5484] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 reported cstate change: config changed from index 6 to 7, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New cstate: current_term: 1 leader_uuid: "aad81047009a4ffcbc450b60030583b5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:13:04.717397 27761 raft_consensus.cc:2932] T 95cb5060152e4237988a4c5723a62601 P 6331f77dae9d452184e8cd0cbf5f1b43 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } } }
I20240501 08:13:04.726135 30916 tablet_service.cc:1514] Processing DeleteTablet for tablet 95cb5060152e4237988a4c5723a62601 with delete_type TABLET_DATA_TOMBSTONED (TS a29cd590999b4f32849ff5689cd53c11 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54036
I20240501 08:13:04.740944 4244 tablet_replica.cc:330] T 95cb5060152e4237988a4c5723a62601 P a29cd590999b4f32849ff5689cd53c11: stopping tablet replica
I20240501 08:13:04.741568 4244 raft_consensus.cc:2229] T 95cb5060152e4237988a4c5723a62601 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:04.741974 4244 raft_consensus.cc:2258] T 95cb5060152e4237988a4c5723a62601 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:04.742919 4244 ts_tablet_manager.cc:1875] T 95cb5060152e4237988a4c5723a62601 P a29cd590999b4f32849ff5689cd53c11: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:04.751892 4244 ts_tablet_manager.cc:1888] T 95cb5060152e4237988a4c5723a62601 P a29cd590999b4f32849ff5689cd53c11: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 08:13:04.752118 4244 log.cc:1196] T 95cb5060152e4237988a4c5723a62601 P a29cd590999b4f32849ff5689cd53c11: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/wals/95cb5060152e4237988a4c5723a62601
I20240501 08:13:04.752959 27606 catalog_manager.cc:4830] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): tablet 95cb5060152e4237988a4c5723a62601 (table rebalancer_extra_table_34 [id=1d72487b1d1a4310bbfd3ad3afddbee9]) successfully deleted
I20240501 08:13:04.772986 4127 raft_consensus.cc:1050] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6: attempting to promote NON_VOTER aad81047009a4ffcbc450b60030583b5 to VOTER
I20240501 08:13:04.773808 4127 consensus_queue.cc:237] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } }
I20240501 08:13:04.776448 27889 raft_consensus.cc:1261] T 7affd631d1e444eaaf8a4c51267a204f P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:13:04.776978 28020 raft_consensus.cc:1261] T 7affd631d1e444eaaf8a4c51267a204f P aad81047009a4ffcbc450b60030583b5 [term 1 LEARNER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:13:04.777479 4130 consensus_queue.cc:1035] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:13:04.778252 4130 consensus_queue.cc:1035] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:13:04.788802 30934 raft_consensus.cc:1261] T 7affd631d1e444eaaf8a4c51267a204f P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:13:04.789472 4253 consensus_queue.cc:1035] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:13:04.790467 4130 raft_consensus.cc:2932] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:13:04.794447 27892 raft_consensus.cc:2932] T 7affd631d1e444eaaf8a4c51267a204f P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:13:04.795450 30937 raft_consensus.cc:2932] T 7affd631d1e444eaaf8a4c51267a204f P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:13:04.803172 27612 catalog_manager.cc:5484] T 7affd631d1e444eaaf8a4c51267a204f P a29cd590999b4f32849ff5689cd53c11 reported cstate change: config changed from index 5 to 6, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:13:04.814525 28020 raft_consensus.cc:2932] T 7affd631d1e444eaaf8a4c51267a204f P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aad81047009a4ffcbc450b60030583b5 (127.21.84.131) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { replace: true } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:13:04.825906 30533 consensus_queue.cc:237] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } }
I20240501 08:13:04.827785 28020 raft_consensus.cc:1261] T 7affd631d1e444eaaf8a4c51267a204f P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:13:04.828354 27892 raft_consensus.cc:1261] T 7affd631d1e444eaaf8a4c51267a204f P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 08:13:04.828364 4253 consensus_queue.cc:1035] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240501 08:13:04.829355 4127 consensus_queue.cc:1035] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:13:04.832222 4127 raft_consensus.cc:2932] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:13:04.834021 27892 raft_consensus.cc:2932] T 7affd631d1e444eaaf8a4c51267a204f P f1525cbc82a24d13b65f136695c903af [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:13:04.835628 27607 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7affd631d1e444eaaf8a4c51267a204f with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 08:13:04.837615 28020 raft_consensus.cc:2932] T 7affd631d1e444eaaf8a4c51267a204f P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } }
I20240501 08:13:04.839876 27612 catalog_manager.cc:5484] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 reported cstate change: config changed from index 6 to 7, VOTER a29cd590999b4f32849ff5689cd53c11 (127.21.84.136) evicted. New cstate: current_term: 1 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:13:04.844597 30916 tablet_service.cc:1514] Processing DeleteTablet for tablet 7affd631d1e444eaaf8a4c51267a204f with delete_type TABLET_DATA_TOMBSTONED (TS a29cd590999b4f32849ff5689cd53c11 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54036
I20240501 08:13:04.844956 4244 tablet_replica.cc:330] T 7affd631d1e444eaaf8a4c51267a204f P a29cd590999b4f32849ff5689cd53c11: stopping tablet replica
I20240501 08:13:04.845155 4244 raft_consensus.cc:2229] T 7affd631d1e444eaaf8a4c51267a204f P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:04.845410 4244 raft_consensus.cc:2258] T 7affd631d1e444eaaf8a4c51267a204f P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:04.850147 4244 ts_tablet_manager.cc:1875] T 7affd631d1e444eaaf8a4c51267a204f P a29cd590999b4f32849ff5689cd53c11: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:04.855283 4244 ts_tablet_manager.cc:1888] T 7affd631d1e444eaaf8a4c51267a204f P a29cd590999b4f32849ff5689cd53c11: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 08:13:04.855422 4244 log.cc:1196] T 7affd631d1e444eaaf8a4c51267a204f P a29cd590999b4f32849ff5689cd53c11: Deleting WAL directory at /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/wals/7affd631d1e444eaaf8a4c51267a204f
I20240501 08:13:04.856760 27606 catalog_manager.cc:4830] TS a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845): tablet 7affd631d1e444eaaf8a4c51267a204f (table rebalancer_extra_table_54 [id=20ee5c0014e146438233fd009facbf2c]) successfully deleted
W20240501 08:13:05.377775 27696 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.131:36305, user_credentials={real_user=slave}} blocked reactor thread for 50370.7us
W20240501 08:13:05.389431 30606 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.135:33503, user_credentials={real_user=slave}} blocked reactor thread for 153914us
W20240501 08:13:05.390797 30737 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.134:39933, user_credentials={real_user=slave}} blocked reactor thread for 54530.2us
I20240501 08:13:05.532738 30514 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240501 08:13:05.584004 27826 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.135:33503, user_credentials={real_user=slave}} blocked reactor thread for 54662.5us
I20240501 08:13:05.584298 27741 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:13:05.618512 30784 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:05.668992 28131 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 08:13:05.725467 27871 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 08:13:05.733033 30650 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:05.734721 28002 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 08:13:05.857323 30870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.134:39933, user_credentials={real_user=slave}} blocked reactor thread for 50675.4us
I20240501 08:13:05.866762 30916 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:06.362601 30514 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:13:06.400372 30650 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:06.418210 27741 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:13:06.514407 28131 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 08:13:06.548447 30916 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:06.560520 28002 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:06.599296 27871 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 08:13:06.604439 30784 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:07.971443 30514 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:13:08.000670 27741 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:13:08.026072 30784 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:08.074944 27871 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 08:13:08.075227 30916 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:08.093291 28002 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:08.110774 28131 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 08:13:08.141474 30650 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:08.801062 30784 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:08.834461 30650 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:08.834443 30514 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:13:08.869104 27741 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:13:08.912982 28131 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 08:13:08.926119 30916 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:08.952687 27871 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 08:13:08.976820 28002 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 08:13:09.653422 27697 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.133:38293, user_credentials={real_user=slave}} blocked reactor thread for 175103us
W20240501 08:13:09.741266 28085 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.134:39933, user_credentials={real_user=slave}} blocked reactor thread for 282424us
W20240501 08:13:09.765043 30472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.136:46845, user_credentials={real_user=slave}} blocked reactor thread for 58463.1us
W20240501 08:13:09.799183 30737 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.132:41323, user_credentials={real_user=slave}} blocked reactor thread for 47902us
W20240501 08:13:09.828265 27825 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.132:41323, user_credentials={real_user=slave}} blocked reactor thread for 78908.1us
I20240501 08:13:09.952014 27741 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:13:09.995370 30784 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:09.999693 30514 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:13:10.049732 30650 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:10.096983 28131 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 08:13:10.131697 27871 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 08:13:10.134124 30916 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:10.176757 28002 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 08:13:10.767302 27958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.130:36503, user_credentials={real_user=slave}} blocked reactor thread for 49049.5us
I20240501 08:13:11.059204 28131 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 08:13:11.059900 30784 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:11.094162 30514 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:13:11.106189 28002 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:11.123210 30916 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:11.135044 27871 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 08:13:11.156700 30650 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:11.210033 27741 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
30c0f17c356e49fca6e933d80d164d83 | 127.21.84.190:36935 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.21.84.148:39605 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
31811708c6774037b82df44024b229b6 | 127.21.84.133:38293 | HEALTHY | <none> | 11 | 0
6331f77dae9d452184e8cd0cbf5f1b43 | 127.21.84.129:43471 | HEALTHY | <none> | 21 | 0
64998323a1f4482d8704bf20594d13a5 | 127.21.84.135:33503 | HEALTHY | <none> | 17 | 0
91d11eae5ef6432ca86eb8ed446b6ea6 | 127.21.84.132:41323 | HEALTHY | <none> | 16 | 0
9a022e1a501f4d44a0468054d6c3de47 | 127.21.84.134:39933 | HEALTHY | <none> | 9 | 0
a29cd590999b4f32849ff5689cd53c11 | 127.21.84.136:46845 | HEALTHY | <none> | 20 | 0
aad81047009a4ffcbc450b60030583b5 | 127.21.84.131:36305 | HEALTHY | <none> | 13 | 0
f1525cbc82a24d13b65f136695c903af | 127.21.84.130:36503 | HEALTHY | <none> | 19 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.21.84.129 | experimental | 127.21.84.129:43471
local_ip_for_outbound_sockets | 127.21.84.130 | experimental | 127.21.84.130:36503
local_ip_for_outbound_sockets | 127.21.84.131 | experimental | 127.21.84.131:36305
local_ip_for_outbound_sockets | 127.21.84.132 | experimental | 127.21.84.132:41323
local_ip_for_outbound_sockets | 127.21.84.133 | experimental | 127.21.84.133:38293
local_ip_for_outbound_sockets | 127.21.84.134 | experimental | 127.21.84.134:39933
local_ip_for_outbound_sockets | 127.21.84.135 | experimental | 127.21.84.135:33503
local_ip_for_outbound_sockets | 127.21.84.136 | experimental | 127.21.84.136:46845
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.21.84.129:43471
server_dump_info_path | /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.21.84.130:36503
server_dump_info_path | /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.21.84.131:36305
server_dump_info_path | /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.21.84.132:41323
server_dump_info_path | /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.21.84.133:38293
server_dump_info_path | /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.21.84.134:39933
server_dump_info_path | /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.21.84.135:33503
server_dump_info_path | /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.21.84.136:46845
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.21.84.148:39605 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------------------+----+---------+---------------+---------+------------+------------------+-------------
rebalance_test_table_1 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_11 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_13 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_15 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_17 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_19 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_5 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_7 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_9 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalancer_extra_table_0 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalancer_extra_table_10 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalancer_extra_table_12 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalancer_extra_table_14 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalancer_extra_table_16 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalancer_extra_table_18 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalancer_extra_table_2 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalancer_extra_table_20 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalancer_extra_table_22 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalancer_extra_table_24 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalancer_extra_table_26 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalancer_extra_table_28 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalancer_extra_table_30 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalancer_extra_table_32 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalancer_extra_table_34 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalancer_extra_table_36 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalancer_extra_table_38 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalancer_extra_table_4 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalancer_extra_table_40 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalancer_extra_table_42 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalancer_extra_table_44 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalancer_extra_table_46 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalancer_extra_table_48 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalancer_extra_table_50 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalancer_extra_table_52 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalancer_extra_table_54 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalancer_extra_table_56 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalancer_extra_table_58 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalancer_extra_table_6 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalancer_extra_table_60 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalancer_extra_table_62 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalancer_extra_table_8 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 47
First Quartile | 47
Median | 47
Third Quartile | 48
Maximum | 48
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 42
Tablets | 126
Replicas | 378
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 08:13:11.380117 21842 log_verifier.cc:126] Checking tablet 00205a461b704a19bbd39bc71361e529
I20240501 08:13:11.410938 21842 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 00205a461b704a19bbd39bc71361e529
I20240501 08:13:11.411087 21842 log_verifier.cc:126] Checking tablet 00a706ff4ea7456aacd2586791704e35
I20240501 08:13:11.445276 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 00a706ff4ea7456aacd2586791704e35
I20240501 08:13:11.445430 21842 log_verifier.cc:126] Checking tablet 00a73d23b12d48a2966f94789db4369a
I20240501 08:13:11.477141 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 00a73d23b12d48a2966f94789db4369a
I20240501 08:13:11.477304 21842 log_verifier.cc:126] Checking tablet 019d8108d6d84d608fd8d5237f3f20ff
I20240501 08:13:11.507304 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 019d8108d6d84d608fd8d5237f3f20ff
I20240501 08:13:11.507452 21842 log_verifier.cc:126] Checking tablet 0481654b2e3c45e98fe51845319c25ee
I20240501 08:13:11.535928 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 0481654b2e3c45e98fe51845319c25ee
I20240501 08:13:11.536051 21842 log_verifier.cc:126] Checking tablet 05b2485fa01a4706a85ebc150306637f
I20240501 08:13:11.581727 21842 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 05b2485fa01a4706a85ebc150306637f
I20240501 08:13:11.581881 21842 log_verifier.cc:126] Checking tablet 05eac5f1b08945f8b7b8c78d0f4d3d3a
I20240501 08:13:11.611335 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 05eac5f1b08945f8b7b8c78d0f4d3d3a
I20240501 08:13:11.611481 21842 log_verifier.cc:126] Checking tablet 0ac9368b897246ac8b2f6b2f096ff3ea
I20240501 08:13:11.639289 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 0ac9368b897246ac8b2f6b2f096ff3ea
I20240501 08:13:11.639426 21842 log_verifier.cc:126] Checking tablet 0be6dad1a409426c9a3dc0866505b3ea
I20240501 08:13:11.681066 21842 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 0be6dad1a409426c9a3dc0866505b3ea
I20240501 08:13:11.681210 21842 log_verifier.cc:126] Checking tablet 0e06cf862c954e8fb1ca087ec5184497
I20240501 08:13:11.713584 21842 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 0e06cf862c954e8fb1ca087ec5184497
I20240501 08:13:11.713739 21842 log_verifier.cc:126] Checking tablet 0e0f87fa9df24b43bedf063a9063bcc5
I20240501 08:13:11.771024 21842 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 0e0f87fa9df24b43bedf063a9063bcc5
I20240501 08:13:11.771162 21842 log_verifier.cc:126] Checking tablet 103e1394bfe34a86ba1d9487d75fd512
I20240501 08:13:11.801651 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 103e1394bfe34a86ba1d9487d75fd512
I20240501 08:13:11.801792 21842 log_verifier.cc:126] Checking tablet 131eac1e19264e00b76500d2ba42697a
I20240501 08:13:11.831655 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 131eac1e19264e00b76500d2ba42697a
I20240501 08:13:11.831795 21842 log_verifier.cc:126] Checking tablet 132d1b9866574dcea3beed5835e4a22f
I20240501 08:13:11.879846 21842 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 132d1b9866574dcea3beed5835e4a22f
I20240501 08:13:11.879998 21842 log_verifier.cc:126] Checking tablet 1448b8f2d5844ad7aec9bcd3874c5c01
I20240501 08:13:11.914850 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 1448b8f2d5844ad7aec9bcd3874c5c01
I20240501 08:13:11.915014 21842 log_verifier.cc:126] Checking tablet 19f899c9c362457a9007aa19ec685fd0
I20240501 08:13:11.944917 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 19f899c9c362457a9007aa19ec685fd0
I20240501 08:13:11.945048 21842 log_verifier.cc:126] Checking tablet 220add4033694ff381e023324a2a8b02
I20240501 08:13:11.978132 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 220add4033694ff381e023324a2a8b02
I20240501 08:13:11.978296 21842 log_verifier.cc:126] Checking tablet 22cdd207ff0c49e0b89d68de068d3603
I20240501 08:13:12.008051 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 22cdd207ff0c49e0b89d68de068d3603
I20240501 08:13:12.008203 21842 log_verifier.cc:126] Checking tablet 24825c85238b4d618c65d00107135420
I20240501 08:13:12.056562 21842 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 24825c85238b4d618c65d00107135420
I20240501 08:13:12.056704 21842 log_verifier.cc:126] Checking tablet 2518578063474b668508d644a2098331
I20240501 08:13:12.085536 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 2518578063474b668508d644a2098331
I20240501 08:13:12.085654 21842 log_verifier.cc:126] Checking tablet 287ebcf5c8af4469af28e69b6008af0a
I20240501 08:13:12.114485 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 287ebcf5c8af4469af28e69b6008af0a
I20240501 08:13:12.114622 21842 log_verifier.cc:126] Checking tablet 28a1f9ac749143de951c1b41beec9875
I20240501 08:13:12.146540 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 28a1f9ac749143de951c1b41beec9875
I20240501 08:13:12.146673 21842 log_verifier.cc:126] Checking tablet 28a7c90dfdf4459aae99c552cfba5b4b
I20240501 08:13:12.177611 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 28a7c90dfdf4459aae99c552cfba5b4b
I20240501 08:13:12.177774 21842 log_verifier.cc:126] Checking tablet 2ce43dc81a1b42c3a63124490d8e60cd
I20240501 08:13:12.206007 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 2ce43dc81a1b42c3a63124490d8e60cd
I20240501 08:13:12.206140 21842 log_verifier.cc:126] Checking tablet 2e1dd7a48a574ffcbea936af5b0fe192
I20240501 08:13:12.242697 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 2e1dd7a48a574ffcbea936af5b0fe192
I20240501 08:13:12.242831 21842 log_verifier.cc:126] Checking tablet 30130709805e4fa6ae94a46034710863
I20240501 08:13:12.293506 21842 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 30130709805e4fa6ae94a46034710863
I20240501 08:13:12.293617 21842 log_verifier.cc:126] Checking tablet 325b3dbba80d491dab81840e8eb171e9
I20240501 08:13:12.325541 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 325b3dbba80d491dab81840e8eb171e9
I20240501 08:13:12.325702 21842 log_verifier.cc:126] Checking tablet 32a54b70dcc447828ec293a9ff0afb80
I20240501 08:13:12.386914 21842 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 32a54b70dcc447828ec293a9ff0afb80
I20240501 08:13:12.387059 21842 log_verifier.cc:126] Checking tablet 367847f920184dd6abcdcd7564f76376
I20240501 08:13:12.440896 21842 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 367847f920184dd6abcdcd7564f76376
I20240501 08:13:12.441057 21842 log_verifier.cc:126] Checking tablet 38c47b76fb304221b60d2b572485ad9f
I20240501 08:13:12.473620 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 38c47b76fb304221b60d2b572485ad9f
I20240501 08:13:12.473747 21842 log_verifier.cc:126] Checking tablet 3aa09c722530431384e5e7ee9759230c
I20240501 08:13:12.505709 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 3aa09c722530431384e5e7ee9759230c
I20240501 08:13:12.505852 21842 log_verifier.cc:126] Checking tablet 40a4b09831f84433b6dc126b8445cf41
I20240501 08:13:12.537014 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 40a4b09831f84433b6dc126b8445cf41
I20240501 08:13:12.537135 21842 log_verifier.cc:126] Checking tablet 448a24d99bbe4fe4a571dc49a04ee275
I20240501 08:13:12.570399 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 448a24d99bbe4fe4a571dc49a04ee275
I20240501 08:13:12.570560 21842 log_verifier.cc:126] Checking tablet 462c892043d8475c99389c6580a66f0c
I20240501 08:13:12.608271 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 462c892043d8475c99389c6580a66f0c
I20240501 08:13:12.608425 21842 log_verifier.cc:126] Checking tablet 489db424748b4a73966b047e81f36c83
I20240501 08:13:12.643970 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 489db424748b4a73966b047e81f36c83
I20240501 08:13:12.644086 21842 log_verifier.cc:126] Checking tablet 4973818ff02b4b7eba922b7fe9193ca6
I20240501 08:13:12.672816 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 4973818ff02b4b7eba922b7fe9193ca6
I20240501 08:13:12.672937 21842 log_verifier.cc:126] Checking tablet 4b3e3d39f82549079293d98357884bf0
I20240501 08:13:12.726874 21842 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 4b3e3d39f82549079293d98357884bf0
I20240501 08:13:12.727051 21842 log_verifier.cc:126] Checking tablet 4e58bc2dffad423f942ce50144030da4
I20240501 08:13:12.763115 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 4e58bc2dffad423f942ce50144030da4
I20240501 08:13:12.763255 21842 log_verifier.cc:126] Checking tablet 4f8c0a8e86f04f32b79743f819948544
I20240501 08:13:12.795329 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 4f8c0a8e86f04f32b79743f819948544
I20240501 08:13:12.795470 21842 log_verifier.cc:126] Checking tablet 4fd595c79cdc43f9a8afadb6b548bf64
I20240501 08:13:12.848804 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 4fd595c79cdc43f9a8afadb6b548bf64
I20240501 08:13:12.848933 21842 log_verifier.cc:126] Checking tablet 52b69ecbe4094f32a1ed73b71b934451
I20240501 08:13:12.897321 21842 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 52b69ecbe4094f32a1ed73b71b934451
I20240501 08:13:12.897450 21842 log_verifier.cc:126] Checking tablet 5536362915034f3eb86138b45d71b8ad
I20240501 08:13:12.949051 21842 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 5536362915034f3eb86138b45d71b8ad
I20240501 08:13:12.949154 21842 log_verifier.cc:126] Checking tablet 59ad966f760143cf9063e05a0079868c
I20240501 08:13:12.978403 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 59ad966f760143cf9063e05a0079868c
I20240501 08:13:12.978502 21842 log_verifier.cc:126] Checking tablet 5eccb9416d7d438893307f13dd4e73dc
I20240501 08:13:13.028522 21842 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 5eccb9416d7d438893307f13dd4e73dc
I20240501 08:13:13.028657 21842 log_verifier.cc:126] Checking tablet 6137a463366349398eef28755d247e95
I20240501 08:13:13.060519 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 6137a463366349398eef28755d247e95
I20240501 08:13:13.060619 21842 log_verifier.cc:126] Checking tablet 647dba7e192140f9972724bd7e4a2569
I20240501 08:13:13.095798 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 647dba7e192140f9972724bd7e4a2569
I20240501 08:13:13.095914 21842 log_verifier.cc:126] Checking tablet 66b8b9c025b941a1a1b6e681fa0bbe90
I20240501 08:13:13.140965 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 66b8b9c025b941a1a1b6e681fa0bbe90
I20240501 08:13:13.141120 21842 log_verifier.cc:126] Checking tablet 683aa9595f2e456c943fc4d16f22e83c
I20240501 08:13:13.175357 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 683aa9595f2e456c943fc4d16f22e83c
I20240501 08:13:13.175465 21842 log_verifier.cc:126] Checking tablet 6a5f99276dad456ea80e3c9f9618059a
I20240501 08:13:13.207651 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 6a5f99276dad456ea80e3c9f9618059a
I20240501 08:13:13.207760 21842 log_verifier.cc:126] Checking tablet 6c04880ea87d4894bd3a63d94f4a69f4
I20240501 08:13:13.263752 21842 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 6c04880ea87d4894bd3a63d94f4a69f4
I20240501 08:13:13.263865 21842 log_verifier.cc:126] Checking tablet 6c08d494131240c1ae8180f652208325
I20240501 08:13:13.291419 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 6c08d494131240c1ae8180f652208325
I20240501 08:13:13.291538 21842 log_verifier.cc:126] Checking tablet 6c324b1136ce429290d7286399bb7121
I20240501 08:13:13.324381 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 6c324b1136ce429290d7286399bb7121
I20240501 08:13:13.324510 21842 log_verifier.cc:126] Checking tablet 6c9bb7ab554e4023892339cd9e0c2a75
I20240501 08:13:13.360424 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 6c9bb7ab554e4023892339cd9e0c2a75
I20240501 08:13:13.360544 21842 log_verifier.cc:126] Checking tablet 728616fd583a4afea00e8fe52880e55b
I20240501 08:13:13.446272 21842 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 728616fd583a4afea00e8fe52880e55b
I20240501 08:13:13.446375 21842 log_verifier.cc:126] Checking tablet 73058afba1214e21b655283c0612a0ec
I20240501 08:13:13.480867 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 73058afba1214e21b655283c0612a0ec
I20240501 08:13:13.480986 21842 log_verifier.cc:126] Checking tablet 73f40a7ef6954b34a48d8c5aeeaebc0a
I20240501 08:13:13.517725 21842 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 73f40a7ef6954b34a48d8c5aeeaebc0a
I20240501 08:13:13.517817 21842 log_verifier.cc:126] Checking tablet 77513d2aea9246b4b37ab54e2a6b496a
I20240501 08:13:13.546077 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 77513d2aea9246b4b37ab54e2a6b496a
I20240501 08:13:13.546171 21842 log_verifier.cc:126] Checking tablet 78ecdb4eacb5472eaa80273f58f7e123
I20240501 08:13:13.575026 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 78ecdb4eacb5472eaa80273f58f7e123
I20240501 08:13:13.575126 21842 log_verifier.cc:126] Checking tablet 79bd880115e1444594881c95c55016d2
I20240501 08:13:13.604681 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 79bd880115e1444594881c95c55016d2
I20240501 08:13:13.604789 21842 log_verifier.cc:126] Checking tablet 7affd631d1e444eaaf8a4c51267a204f
I20240501 08:13:13.638005 21842 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 7affd631d1e444eaaf8a4c51267a204f
I20240501 08:13:13.638108 21842 log_verifier.cc:126] Checking tablet 7c3b2b31a9e747c69ff67969d878bc64
I20240501 08:13:13.668231 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 7c3b2b31a9e747c69ff67969d878bc64
I20240501 08:13:13.668331 21842 log_verifier.cc:126] Checking tablet 7f7211c8c0e041bf89930aa50a4438ea
I20240501 08:13:13.740906 21842 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 7f7211c8c0e041bf89930aa50a4438ea
I20240501 08:13:13.741034 21842 log_verifier.cc:126] Checking tablet 802538381de843ab8d326014a0b4ec15
I20240501 08:13:13.787444 21842 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 802538381de843ab8d326014a0b4ec15
I20240501 08:13:13.787564 21842 log_verifier.cc:126] Checking tablet 80bb0e53796f4b319fa9f877451076f3
I20240501 08:13:13.817143 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 80bb0e53796f4b319fa9f877451076f3
I20240501 08:13:13.817265 21842 log_verifier.cc:126] Checking tablet 8277d70d49f54197af60e0be86d34a9b
I20240501 08:13:13.846457 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 8277d70d49f54197af60e0be86d34a9b
I20240501 08:13:13.846580 21842 log_verifier.cc:126] Checking tablet 82d6e5e399744b96a66373240ce1764d
I20240501 08:13:13.876469 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 82d6e5e399744b96a66373240ce1764d
I20240501 08:13:13.876561 21842 log_verifier.cc:126] Checking tablet 83b422d7436a41e4aa6398963fb40f1e
I20240501 08:13:13.904781 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 83b422d7436a41e4aa6398963fb40f1e
I20240501 08:13:13.904898 21842 log_verifier.cc:126] Checking tablet 87804550560d45269e6d4d6fe8938f55
I20240501 08:13:13.939956 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 87804550560d45269e6d4d6fe8938f55
I20240501 08:13:13.940084 21842 log_verifier.cc:126] Checking tablet 87bd346903dc4c2c85ecaf362fe57799
I20240501 08:13:13.974742 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 87bd346903dc4c2c85ecaf362fe57799
I20240501 08:13:13.974860 21842 log_verifier.cc:126] Checking tablet 893c1555ca0849da8f48a4f253b42153
I20240501 08:13:14.016597 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 893c1555ca0849da8f48a4f253b42153
I20240501 08:13:14.016698 21842 log_verifier.cc:126] Checking tablet 8ab0410a98974002ab68dd0f6a456d59
I20240501 08:13:14.045300 21842 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 8ab0410a98974002ab68dd0f6a456d59
I20240501 08:13:14.045418 21842 log_verifier.cc:126] Checking tablet 8bbd50f618d14aad8e45c2aa6fbabc1f
I20240501 08:13:14.075611 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 8bbd50f618d14aad8e45c2aa6fbabc1f
I20240501 08:13:14.075701 21842 log_verifier.cc:126] Checking tablet 8c3d0b4ae0404bdfada8a86b23db9d30
I20240501 08:13:14.103209 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 8c3d0b4ae0404bdfada8a86b23db9d30
I20240501 08:13:14.103296 21842 log_verifier.cc:126] Checking tablet 8eb9763a1573455287b7137fe7fecf62
I20240501 08:13:14.155498 21842 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 8eb9763a1573455287b7137fe7fecf62
I20240501 08:13:14.155603 21842 log_verifier.cc:126] Checking tablet 928b70089c234425862c033341c82bad
I20240501 08:13:14.185019 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 928b70089c234425862c033341c82bad
I20240501 08:13:14.185130 21842 log_verifier.cc:126] Checking tablet 946add184f4a46f28fd631e8a3bc0ba7
I20240501 08:13:14.215004 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 946add184f4a46f28fd631e8a3bc0ba7
I20240501 08:13:14.215193 21842 log_verifier.cc:126] Checking tablet 948f6afb45594870bfe44b45d303f88f
I20240501 08:13:14.244644 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 948f6afb45594870bfe44b45d303f88f
I20240501 08:13:14.244763 21842 log_verifier.cc:126] Checking tablet 95cb5060152e4237988a4c5723a62601
I20240501 08:13:14.281471 21842 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 95cb5060152e4237988a4c5723a62601
I20240501 08:13:14.281569 21842 log_verifier.cc:126] Checking tablet 9820951bffc749139229e2b79c8ab597
I20240501 08:13:14.328262 21842 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 9820951bffc749139229e2b79c8ab597
I20240501 08:13:14.328431 21842 log_verifier.cc:126] Checking tablet 98525304f2334bf0a3f52c06df58b3ba
I20240501 08:13:14.357668 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 98525304f2334bf0a3f52c06df58b3ba
I20240501 08:13:14.357837 21842 log_verifier.cc:126] Checking tablet 98c8e3a6f66a42b08507668f7c3a8af1
I20240501 08:13:14.388536 21842 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 98c8e3a6f66a42b08507668f7c3a8af1
I20240501 08:13:14.388646 21842 log_verifier.cc:126] Checking tablet 9b5b0b1e46d14d78806859541e0ab3b6
I20240501 08:13:14.417282 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 9b5b0b1e46d14d78806859541e0ab3b6
I20240501 08:13:14.417476 21842 log_verifier.cc:126] Checking tablet 9df818d6cb4946989f8a4d2e2ce9b8bb
I20240501 08:13:14.450896 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 9df818d6cb4946989f8a4d2e2ce9b8bb
I20240501 08:13:14.451187 21842 log_verifier.cc:126] Checking tablet 9e1b0cfacf5148a5998c356f1c6f006f
I20240501 08:13:14.480619 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 9e1b0cfacf5148a5998c356f1c6f006f
I20240501 08:13:14.480815 21842 log_verifier.cc:126] Checking tablet 9e22cda990da493d9aa7d9401552cb6e
I20240501 08:13:14.513115 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 9e22cda990da493d9aa7d9401552cb6e
I20240501 08:13:14.513213 21842 log_verifier.cc:126] Checking tablet 9eecbc3985624f3889d8e554cd585b11
I20240501 08:13:14.537261 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 9eecbc3985624f3889d8e554cd585b11
I20240501 08:13:14.537405 21842 log_verifier.cc:126] Checking tablet a05223cb2f20478b8d29ffa065b4c67b
I20240501 08:13:14.579952 21842 log_verifier.cc:177] Verified matching terms for 7 ops in tablet a05223cb2f20478b8d29ffa065b4c67b
I20240501 08:13:14.580081 21842 log_verifier.cc:126] Checking tablet a0f992f08fb841a5b18b14f5ad0b9c58
I20240501 08:13:14.625573 21842 log_verifier.cc:177] Verified matching terms for 6 ops in tablet a0f992f08fb841a5b18b14f5ad0b9c58
I20240501 08:13:14.625687 21842 log_verifier.cc:126] Checking tablet a1965bd993d748f6a33cf93e8a3d70bf
I20240501 08:13:14.652563 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet a1965bd993d748f6a33cf93e8a3d70bf
I20240501 08:13:14.652683 21842 log_verifier.cc:126] Checking tablet a1fde6b765374df9bc4674a5d678c920
I20240501 08:13:14.680317 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet a1fde6b765374df9bc4674a5d678c920
I20240501 08:13:14.680424 21842 log_verifier.cc:126] Checking tablet a2981c0863bc41729605f8b076b6a6cb
I20240501 08:13:14.723052 21842 log_verifier.cc:177] Verified matching terms for 6 ops in tablet a2981c0863bc41729605f8b076b6a6cb
I20240501 08:13:14.723166 21842 log_verifier.cc:126] Checking tablet a3f0322a212a459b925a0f4e2dd2457e
I20240501 08:13:14.755450 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet a3f0322a212a459b925a0f4e2dd2457e
I20240501 08:13:14.755535 21842 log_verifier.cc:126] Checking tablet ac4914417d9e44e7a9591be5386f1d8b
I20240501 08:13:14.798640 21842 log_verifier.cc:177] Verified matching terms for 6 ops in tablet ac4914417d9e44e7a9591be5386f1d8b
I20240501 08:13:14.798749 21842 log_verifier.cc:126] Checking tablet b0906abad5c14912b94e9c017d2ecf8d
I20240501 08:13:14.825603 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet b0906abad5c14912b94e9c017d2ecf8d
I20240501 08:13:14.825707 21842 log_verifier.cc:126] Checking tablet b0eb3ba3e6e347ebaba87323f01cd8d4
I20240501 08:13:14.852306 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet b0eb3ba3e6e347ebaba87323f01cd8d4
I20240501 08:13:14.852391 21842 log_verifier.cc:126] Checking tablet b12559fc416a4f3cbffa018611226bcc
I20240501 08:13:14.897068 21842 log_verifier.cc:177] Verified matching terms for 9 ops in tablet b12559fc416a4f3cbffa018611226bcc
I20240501 08:13:14.897188 21842 log_verifier.cc:126] Checking tablet b16391636d5f4ec48c9e4f18ed40badd
I20240501 08:13:14.926718 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet b16391636d5f4ec48c9e4f18ed40badd
I20240501 08:13:14.926816 21842 log_verifier.cc:126] Checking tablet b5d7223054aa4308917fd402d65ded5d
I20240501 08:13:14.954386 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet b5d7223054aa4308917fd402d65ded5d
I20240501 08:13:14.954504 21842 log_verifier.cc:126] Checking tablet b605254185d140d8ab90fcaa5229f168
I20240501 08:13:14.984799 21842 log_verifier.cc:177] Verified matching terms for 5 ops in tablet b605254185d140d8ab90fcaa5229f168
I20240501 08:13:14.984898 21842 log_verifier.cc:126] Checking tablet b6ad187384514267b57cc994c9612709
I20240501 08:13:15.017517 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet b6ad187384514267b57cc994c9612709
I20240501 08:13:15.017606 21842 log_verifier.cc:126] Checking tablet b6b0bcb4d07e4125bccf1711a094576e
I20240501 08:13:15.044497 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet b6b0bcb4d07e4125bccf1711a094576e
I20240501 08:13:15.044615 21842 log_verifier.cc:126] Checking tablet b82bf9ba2a7141cc81c6767cdc774c3a
I20240501 08:13:15.070065 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet b82bf9ba2a7141cc81c6767cdc774c3a
I20240501 08:13:15.070165 21842 log_verifier.cc:126] Checking tablet bb7dae019c7e4842b0daa35ee4b40367
I20240501 08:13:15.111153 21842 log_verifier.cc:177] Verified matching terms for 6 ops in tablet bb7dae019c7e4842b0daa35ee4b40367
I20240501 08:13:15.111244 21842 log_verifier.cc:126] Checking tablet bc5778711a4c479881088af790f5b7d5
I20240501 08:13:15.152953 21842 log_verifier.cc:177] Verified matching terms for 9 ops in tablet bc5778711a4c479881088af790f5b7d5
I20240501 08:13:15.153069 21842 log_verifier.cc:126] Checking tablet bc84d1d413994a76bd6cf42335438755
I20240501 08:13:15.182487 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet bc84d1d413994a76bd6cf42335438755
I20240501 08:13:15.182651 21842 log_verifier.cc:126] Checking tablet c0e30c62c2d441618195f30f15647487
I20240501 08:13:15.218003 21842 log_verifier.cc:177] Verified matching terms for 5 ops in tablet c0e30c62c2d441618195f30f15647487
I20240501 08:13:15.218164 21842 log_verifier.cc:126] Checking tablet c4851258508d46c88493503a6257f365
I20240501 08:13:15.259742 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet c4851258508d46c88493503a6257f365
I20240501 08:13:15.259857 21842 log_verifier.cc:126] Checking tablet c51625843b8d44a8879fa2fdbecbee5f
I20240501 08:13:15.291369 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet c51625843b8d44a8879fa2fdbecbee5f
I20240501 08:13:15.291554 21842 log_verifier.cc:126] Checking tablet c5f4c78e69fb4c67ad65a7b0c1466798
I20240501 08:13:15.327775 21842 log_verifier.cc:177] Verified matching terms for 5 ops in tablet c5f4c78e69fb4c67ad65a7b0c1466798
I20240501 08:13:15.328006 21842 log_verifier.cc:126] Checking tablet c832fea33e4744868b69e6c29e4aea9f
I20240501 08:13:15.359464 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet c832fea33e4744868b69e6c29e4aea9f
I20240501 08:13:15.359627 21842 log_verifier.cc:126] Checking tablet cb6e886e8cdd451d9a192a771482728e
I20240501 08:13:15.391410 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet cb6e886e8cdd451d9a192a771482728e
I20240501 08:13:15.391566 21842 log_verifier.cc:126] Checking tablet d020e1225d5d447f993f920569a05f3d
I20240501 08:13:15.422683 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet d020e1225d5d447f993f920569a05f3d
I20240501 08:13:15.422847 21842 log_verifier.cc:126] Checking tablet d3673fe03fb64c47b30dc2b79042f44a
I20240501 08:13:15.454897 21842 log_verifier.cc:177] Verified matching terms for 5 ops in tablet d3673fe03fb64c47b30dc2b79042f44a
I20240501 08:13:15.455013 21842 log_verifier.cc:126] Checking tablet d6bb70263134414b9945a87087b55a2c
I20240501 08:13:15.485049 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet d6bb70263134414b9945a87087b55a2c
I20240501 08:13:15.485224 21842 log_verifier.cc:126] Checking tablet e12867cd76504438abe91fbe15498230
I20240501 08:13:15.520064 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet e12867cd76504438abe91fbe15498230
I20240501 08:13:15.520148 21842 log_verifier.cc:126] Checking tablet e3d07789edc74dec975e35e9d94cb0ef
I20240501 08:13:15.560082 21842 log_verifier.cc:177] Verified matching terms for 6 ops in tablet e3d07789edc74dec975e35e9d94cb0ef
I20240501 08:13:15.560176 21842 log_verifier.cc:126] Checking tablet e40fb0a0d49f44a1a36c33412c630077
I20240501 08:13:15.605504 21842 log_verifier.cc:177] Verified matching terms for 6 ops in tablet e40fb0a0d49f44a1a36c33412c630077
I20240501 08:13:15.605592 21842 log_verifier.cc:126] Checking tablet e6fb30f10c754d3a8e73e124ec59768b
I20240501 08:13:15.650576 21842 log_verifier.cc:177] Verified matching terms for 9 ops in tablet e6fb30f10c754d3a8e73e124ec59768b
I20240501 08:13:15.650689 21842 log_verifier.cc:126] Checking tablet e94895331d6b474484d987d42f13fd88
I20240501 08:13:15.678720 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet e94895331d6b474484d987d42f13fd88
I20240501 08:13:15.678835 21842 log_verifier.cc:126] Checking tablet ecab521101074a7b80b862642059daf2
I20240501 08:13:15.707254 21842 log_verifier.cc:177] Verified matching terms for 5 ops in tablet ecab521101074a7b80b862642059daf2
I20240501 08:13:15.707355 21842 log_verifier.cc:126] Checking tablet f2baff8656804ce2bfb64a9280e16855
I20240501 08:13:15.750699 21842 log_verifier.cc:177] Verified matching terms for 7 ops in tablet f2baff8656804ce2bfb64a9280e16855
I20240501 08:13:15.750798 21842 log_verifier.cc:126] Checking tablet f3b6691cbe8740d2978c20209c78f3ff
I20240501 08:13:15.805457 21842 log_verifier.cc:177] Verified matching terms for 6 ops in tablet f3b6691cbe8740d2978c20209c78f3ff
I20240501 08:13:15.805558 21842 log_verifier.cc:126] Checking tablet f607d2e086cc4352a05a5ef2b02788cc
I20240501 08:13:15.834074 21842 log_verifier.cc:177] Verified matching terms for 7 ops in tablet f607d2e086cc4352a05a5ef2b02788cc
I20240501 08:13:15.834195 21842 log_verifier.cc:126] Checking tablet f63215605ade41a5bc8d97a35ce33a31
I20240501 08:13:15.864004 21842 log_verifier.cc:177] Verified matching terms for 8 ops in tablet f63215605ade41a5bc8d97a35ce33a31
I20240501 08:13:15.864109 21842 log_verifier.cc:126] Checking tablet fc9e0f35ebcd48b5a27c57b2485e599e
I20240501 08:13:15.895124 21842 log_verifier.cc:177] Verified matching terms for 4 ops in tablet fc9e0f35ebcd48b5a27c57b2485e599e
I20240501 08:13:15.895291 21842 log_verifier.cc:126] Checking tablet fea1af7c0ecc4ddba679a981125d957a
I20240501 08:13:15.922228 21842 log_verifier.cc:177] Verified matching terms for 1 ops in tablet fea1af7c0ecc4ddba679a981125d957a
I20240501 08:13:15.922940 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 27679
I20240501 08:13:15.949283 27801 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:13:16.307420 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 27679
W20240501 08:13:16.365355 30472 connection.cc:535] client connection to 127.21.84.129:43471 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:16.365535 27957 connection.cc:535] client connection to 127.21.84.129:43471 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:16.365536 30873 connection.cc:535] server connection from 127.21.84.129:48739 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:16.365658 30472 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:16.365744 30607 connection.cc:535] client connection to 127.21.84.129:43471 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:16.365772 27957 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:16.365996 30607 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:16.366075 27958 connection.cc:535] server connection from 127.21.84.129:39999 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:16.366393 28087 connection.cc:535] client connection to 127.21.84.129:43471 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:16.366674 28087 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:16.366852 30739 connection.cc:535] client connection to 127.21.84.129:43471 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:16.367100 30739 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:16.372135 30872 connection.cc:535] client connection to 127.21.84.129:43471 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:16.372385 30872 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:13:16.375471 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 27810
W20240501 08:13:16.387523 27957 consensus_peers.cc:485] T 30130709805e4fa6ae94a46034710863 P aad81047009a4ffcbc450b60030583b5 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.388355 27957 consensus_peers.cc:485] T 287ebcf5c8af4469af28e69b6008af0a P aad81047009a4ffcbc450b60030583b5 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:16.389333 27932 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 08:13:16.389551 27957 consensus_peers.cc:485] T 95cb5060152e4237988a4c5723a62601 P aad81047009a4ffcbc450b60030583b5 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.391980 30872 consensus_peers.cc:485] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.392251 30872 consensus_peers.cc:485] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.392407 30872 consensus_peers.cc:485] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.392549 30872 consensus_peers.cc:485] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.392701 30872 consensus_peers.cc:485] T 22cdd207ff0c49e0b89d68de068d3603 P a29cd590999b4f32849ff5689cd53c11 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.403496 28087 consensus_peers.cc:485] T b0eb3ba3e6e347ebaba87323f01cd8d4 P 91d11eae5ef6432ca86eb8ed446b6ea6 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.410269 30472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.129:43471, user_credentials={real_user=slave}} blocked reactor thread for 44675.5us
W20240501 08:13:16.411367 30739 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.129:43471, user_credentials={real_user=slave}} blocked reactor thread for 44337.7us
W20240501 08:13:16.418800 30607 consensus_peers.cc:485] T 00a706ff4ea7456aacd2586791704e35 P 9a022e1a501f4d44a0468054d6c3de47 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.419158 30607 consensus_peers.cc:485] T 683aa9595f2e456c943fc4d16f22e83c P 9a022e1a501f4d44a0468054d6c3de47 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.421577 30607 consensus_peers.cc:485] T 28a1f9ac749143de951c1b41beec9875 P 9a022e1a501f4d44a0468054d6c3de47 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.422014 30607 consensus_peers.cc:485] T c0e30c62c2d441618195f30f15647487 P 9a022e1a501f4d44a0468054d6c3de47 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.422245 30607 consensus_peers.cc:485] T 4973818ff02b4b7eba922b7fe9193ca6 P 9a022e1a501f4d44a0468054d6c3de47 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.425334 30472 consensus_peers.cc:485] T 220add4033694ff381e023324a2a8b02 P 31811708c6774037b82df44024b229b6 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.425490 30472 consensus_peers.cc:485] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.425640 30607 consensus_peers.cc:485] T 462c892043d8475c99389c6580a66f0c P 9a022e1a501f4d44a0468054d6c3de47 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.427533 30607 consensus_peers.cc:485] T 78ecdb4eacb5472eaa80273f58f7e123 P 9a022e1a501f4d44a0468054d6c3de47 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.429318 30472 consensus_peers.cc:485] T e12867cd76504438abe91fbe15498230 P 31811708c6774037b82df44024b229b6 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.430474 30739 consensus_peers.cc:485] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.430609 30739 consensus_peers.cc:485] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.430686 30739 consensus_peers.cc:485] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.538534 30472 consensus_peers.cc:485] T 8c3d0b4ae0404bdfada8a86b23db9d30 P 31811708c6774037b82df44024b229b6 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:16.822192 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 27810
W20240501 08:13:16.879117 30606 connection.cc:535] server connection from 127.21.84.130:57689 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:16.879588 30473 connection.cc:535] client connection to 127.21.84.130:36503 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:16.879621 30740 connection.cc:535] client connection to 127.21.84.130:36503 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:13:16.880030 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 27940
W20240501 08:13:16.880635 27958 connection.cc:535] server connection from 127.21.84.130:59663 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:16.880776 30873 connection.cc:535] client connection to 127.21.84.130:36503 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:16.880807 27958 connection.cc:535] client connection to 127.21.84.130:36503 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:16.881162 28085 connection.cc:535] server connection from 127.21.84.130:41275 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:16.881356 28088 connection.cc:535] client connection to 127.21.84.130:36503 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:16.881569 30608 connection.cc:535] client connection to 127.21.84.130:36503 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:16.882495 30740 consensus_peers.cc:485] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.882608 30740 consensus_peers.cc:485] T 6c9bb7ab554e4023892339cd9e0c2a75 P 64998323a1f4482d8704bf20594d13a5 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.882596 27958 consensus_peers.cc:485] T 287ebcf5c8af4469af28e69b6008af0a P aad81047009a4ffcbc450b60030583b5 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.882668 30740 consensus_peers.cc:485] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.882725 30740 consensus_peers.cc:485] T 647dba7e192140f9972724bd7e4a2569 P 64998323a1f4482d8704bf20594d13a5 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.882779 30740 consensus_peers.cc:485] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.882833 30740 consensus_peers.cc:485] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.882838 30873 consensus_peers.cc:485] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.882896 30740 consensus_peers.cc:485] T fea1af7c0ecc4ddba679a981125d957a P 64998323a1f4482d8704bf20594d13a5 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.883030 30473 consensus_peers.cc:485] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.883165 30473 consensus_peers.cc:485] T 83b422d7436a41e4aa6398963fb40f1e P 31811708c6774037b82df44024b229b6 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.883235 30473 consensus_peers.cc:485] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.883324 30473 consensus_peers.cc:485] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.884092 30608 consensus_peers.cc:485] T c0e30c62c2d441618195f30f15647487 P 9a022e1a501f4d44a0468054d6c3de47 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.885591 28088 consensus_peers.cc:485] T 9e22cda990da493d9aa7d9401552cb6e P 91d11eae5ef6432ca86eb8ed446b6ea6 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.885691 28088 consensus_peers.cc:485] T 5536362915034f3eb86138b45d71b8ad P 91d11eae5ef6432ca86eb8ed446b6ea6 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:16.894184 28062 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:13:16.894534 4740 raft_consensus.cc:482] T e3d07789edc74dec975e35e9d94cb0ef P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:16.894721 4740 raft_consensus.cc:504] T e3d07789edc74dec975e35e9d94cb0ef P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:13:16.895457 4740 leader_election.cc:288] T e3d07789edc74dec975e35e9d94cb0ef P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
W20240501 08:13:16.897114 30739 leader_election.cc:334] T e3d07789edc74dec975e35e9d94cb0ef P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
W20240501 08:13:16.897485 30740 leader_election.cc:334] T e3d07789edc74dec975e35e9d94cb0ef P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
I20240501 08:13:16.897758 30740 leader_election.cc:302] T e3d07789edc74dec975e35e9d94cb0ef P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5; no voters: 6331f77dae9d452184e8cd0cbf5f1b43, f1525cbc82a24d13b65f136695c903af
I20240501 08:13:16.898036 4740 raft_consensus.cc:2726] T e3d07789edc74dec975e35e9d94cb0ef P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:13:16.909198 28088 consensus_peers.cc:485] T 489db424748b4a73966b047e81f36c83 P 91d11eae5ef6432ca86eb8ed446b6ea6 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:16.960712 30873 consensus_peers.cc:485] T a3f0322a212a459b925a0f4e2dd2457e P a29cd590999b4f32849ff5689cd53c11 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:16.969605 4734 raft_consensus.cc:482] T 2518578063474b668508d644a2098331 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:16.969818 4734 raft_consensus.cc:504] T 2518578063474b668508d644a2098331 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:13:16.970772 4734 leader_election.cc:288] T 2518578063474b668508d644a2098331 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:13:16.971261 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2518578063474b668508d644a2098331" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
W20240501 08:13:16.972414 30472 leader_election.cc:334] T 2518578063474b668508d644a2098331 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
I20240501 08:13:16.972564 30472 leader_election.cc:302] T 2518578063474b668508d644a2098331 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31811708c6774037b82df44024b229b6; no voters: 6331f77dae9d452184e8cd0cbf5f1b43, a29cd590999b4f32849ff5689cd53c11
I20240501 08:13:16.972793 4734 raft_consensus.cc:2726] T 2518578063474b668508d644a2098331 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:13:17.006664 4734 raft_consensus.cc:482] T ac4914417d9e44e7a9591be5386f1d8b P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:17.006848 4734 raft_consensus.cc:504] T ac4914417d9e44e7a9591be5386f1d8b P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } }
I20240501 08:13:17.007532 4734 leader_election.cc:288] T ac4914417d9e44e7a9591be5386f1d8b P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
W20240501 08:13:17.009258 30473 leader_election.cc:334] T ac4914417d9e44e7a9591be5386f1d8b P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
W20240501 08:13:17.010120 30472 leader_election.cc:334] T ac4914417d9e44e7a9591be5386f1d8b P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
I20240501 08:13:17.010293 30472 leader_election.cc:302] T ac4914417d9e44e7a9591be5386f1d8b P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31811708c6774037b82df44024b229b6; no voters: 6331f77dae9d452184e8cd0cbf5f1b43, f1525cbc82a24d13b65f136695c903af
I20240501 08:13:17.011034 4734 raft_consensus.cc:2726] T ac4914417d9e44e7a9591be5386f1d8b P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:13:17.046458 30873 consensus_peers.cc:485] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:17.062076 4787 raft_consensus.cc:482] T f3b6691cbe8740d2978c20209c78f3ff P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:17.062305 4787 raft_consensus.cc:504] T f3b6691cbe8740d2978c20209c78f3ff P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:13:17.063145 4787 leader_election.cc:288] T f3b6691cbe8740d2978c20209c78f3ff P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:13:17.063952 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3b6691cbe8740d2978c20209c78f3ff" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
W20240501 08:13:17.065429 28087 leader_election.cc:334] T f3b6691cbe8740d2978c20209c78f3ff P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
I20240501 08:13:17.065569 28087 leader_election.cc:302] T f3b6691cbe8740d2978c20209c78f3ff P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6; no voters: 6331f77dae9d452184e8cd0cbf5f1b43, a29cd590999b4f32849ff5689cd53c11
I20240501 08:13:17.066110 4787 raft_consensus.cc:2726] T f3b6691cbe8740d2978c20209c78f3ff P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:13:17.071332 4774 raft_consensus.cc:482] T bb7dae019c7e4842b0daa35ee4b40367 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:17.071512 4774 raft_consensus.cc:504] T bb7dae019c7e4842b0daa35ee4b40367 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } }
I20240501 08:13:17.072293 4774 leader_election.cc:288] T bb7dae019c7e4842b0daa35ee4b40367 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:13:17.073393 4777 raft_consensus.cc:482] T 9e1b0cfacf5148a5998c356f1c6f006f P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
W20240501 08:13:17.073643 30472 leader_election.cc:334] T bb7dae019c7e4842b0daa35ee4b40367 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
I20240501 08:13:17.073547 4777 raft_consensus.cc:504] T 9e1b0cfacf5148a5998c356f1c6f006f P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:13:17.074250 4777 leader_election.cc:288] T 9e1b0cfacf5148a5998c356f1c6f006f P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:13:17.074544 4774 raft_consensus.cc:482] T 9b5b0b1e46d14d78806859541e0ab3b6 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:17.074666 4774 raft_consensus.cc:504] T 9b5b0b1e46d14d78806859541e0ab3b6 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:13:17.075320 4774 leader_election.cc:288] T 9b5b0b1e46d14d78806859541e0ab3b6 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:13:17.075867 30801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b5b0b1e46d14d78806859541e0ab3b6" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
W20240501 08:13:17.076251 30607 leader_election.cc:334] T 9e1b0cfacf5148a5998c356f1c6f006f P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
I20240501 08:13:17.076622 4777 raft_consensus.cc:482] T 0ac9368b897246ac8b2f6b2f096ff3ea P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:17.076763 4777 raft_consensus.cc:504] T 0ac9368b897246ac8b2f6b2f096ff3ea P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:13:17.077484 4777 leader_election.cc:288] T 0ac9368b897246ac8b2f6b2f096ff3ea P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
W20240501 08:13:17.079816 30472 leader_election.cc:334] T 9b5b0b1e46d14d78806859541e0ab3b6 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
I20240501 08:13:17.079938 30472 leader_election.cc:302] T 9b5b0b1e46d14d78806859541e0ab3b6 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31811708c6774037b82df44024b229b6; no voters: 6331f77dae9d452184e8cd0cbf5f1b43, 64998323a1f4482d8704bf20594d13a5
I20240501 08:13:17.080147 4774 raft_consensus.cc:2726] T 9b5b0b1e46d14d78806859541e0ab3b6 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:13:17.080402 30607 leader_election.cc:334] T 0ac9368b897246ac8b2f6b2f096ff3ea P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
W20240501 08:13:17.085515 30608 leader_election.cc:334] T 9e1b0cfacf5148a5998c356f1c6f006f P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
I20240501 08:13:17.085721 30608 leader_election.cc:302] T 9e1b0cfacf5148a5998c356f1c6f006f P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a022e1a501f4d44a0468054d6c3de47; no voters: 6331f77dae9d452184e8cd0cbf5f1b43, f1525cbc82a24d13b65f136695c903af
I20240501 08:13:17.086072 4777 raft_consensus.cc:2726] T 9e1b0cfacf5148a5998c356f1c6f006f P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:13:17.095683 4787 raft_consensus.cc:482] T f63215605ade41a5bc8d97a35ce33a31 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:17.095875 4787 raft_consensus.cc:504] T f63215605ade41a5bc8d97a35ce33a31 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:13:17.096602 4787 leader_election.cc:288] T f63215605ade41a5bc8d97a35ce33a31 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:13:17.097038 30534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63215605ade41a5bc8d97a35ce33a31" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
W20240501 08:13:17.097888 28087 leader_election.cc:334] T f63215605ade41a5bc8d97a35ce33a31 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
I20240501 08:13:17.098016 28087 leader_election.cc:302] T f63215605ade41a5bc8d97a35ce33a31 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6; no voters: 31811708c6774037b82df44024b229b6, 6331f77dae9d452184e8cd0cbf5f1b43
I20240501 08:13:17.098258 4787 raft_consensus.cc:2726] T f63215605ade41a5bc8d97a35ce33a31 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:13:17.104969 4778 raft_consensus.cc:482] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:17.105154 4778 raft_consensus.cc:504] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:13:17.106072 4778 leader_election.cc:288] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:13:17.106720 30668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66b8b9c025b941a1a1b6e681fa0bbe90" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
W20240501 08:13:17.107420 30739 leader_election.cc:334] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
I20240501 08:13:17.125658 30737 leader_election.cc:302] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5; no voters: 6331f77dae9d452184e8cd0cbf5f1b43, 9a022e1a501f4d44a0468054d6c3de47
I20240501 08:13:17.126039 4725 raft_consensus.cc:2726] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:13:17.147702 4778 raft_consensus.cc:482] T b0906abad5c14912b94e9c017d2ecf8d P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:17.147917 4778 raft_consensus.cc:504] T b0906abad5c14912b94e9c017d2ecf8d P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:13:17.148669 4778 leader_election.cc:288] T b0906abad5c14912b94e9c017d2ecf8d P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
W20240501 08:13:17.150383 30740 leader_election.cc:334] T b0906abad5c14912b94e9c017d2ecf8d P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
W20240501 08:13:17.151206 30739 leader_election.cc:334] T b0906abad5c14912b94e9c017d2ecf8d P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
I20240501 08:13:17.151309 30739 leader_election.cc:302] T b0906abad5c14912b94e9c017d2ecf8d P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5; no voters: 6331f77dae9d452184e8cd0cbf5f1b43, f1525cbc82a24d13b65f136695c903af
I20240501 08:13:17.151492 4778 raft_consensus.cc:2726] T b0906abad5c14912b94e9c017d2ecf8d P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:13:17.157414 4778 raft_consensus.cc:482] T e94895331d6b474484d987d42f13fd88 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:17.157543 4778 raft_consensus.cc:504] T e94895331d6b474484d987d42f13fd88 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:13:17.158138 4778 leader_election.cc:288] T e94895331d6b474484d987d42f13fd88 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:13:17.158561 30668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e94895331d6b474484d987d42f13fd88" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
W20240501 08:13:17.160084 30739 leader_election.cc:334] T e94895331d6b474484d987d42f13fd88 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
I20240501 08:13:17.160188 30739 leader_election.cc:302] T e94895331d6b474484d987d42f13fd88 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5; no voters: 6331f77dae9d452184e8cd0cbf5f1b43, 9a022e1a501f4d44a0468054d6c3de47
I20240501 08:13:17.160431 4725 raft_consensus.cc:2726] T e94895331d6b474484d987d42f13fd88 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:13:17.207335 4778 raft_consensus.cc:482] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:17.207511 4778 raft_consensus.cc:504] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:13:17.208159 4778 leader_election.cc:288] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:13:17.208644 30534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f40a7ef6954b34a48d8c5aeeaebc0a" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
W20240501 08:13:17.209586 30739 leader_election.cc:334] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
I20240501 08:13:17.209712 30739 leader_election.cc:302] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5; no voters: 31811708c6774037b82df44024b229b6, 6331f77dae9d452184e8cd0cbf5f1b43
I20240501 08:13:17.209924 4778 raft_consensus.cc:2726] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:13:17.241300 4787 raft_consensus.cc:482] T 728616fd583a4afea00e8fe52880e55b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:17.241458 4787 raft_consensus.cc:504] T 728616fd583a4afea00e8fe52880e55b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:13:17.242208 4787 leader_election.cc:288] T 728616fd583a4afea00e8fe52880e55b P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:13:17.242668 30801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "728616fd583a4afea00e8fe52880e55b" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
W20240501 08:13:17.243430 28087 leader_election.cc:334] T 728616fd583a4afea00e8fe52880e55b P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
I20240501 08:13:17.243522 28087 leader_election.cc:302] T 728616fd583a4afea00e8fe52880e55b P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6; no voters: 6331f77dae9d452184e8cd0cbf5f1b43, 64998323a1f4482d8704bf20594d13a5
I20240501 08:13:17.243767 4787 raft_consensus.cc:2726] T 728616fd583a4afea00e8fe52880e55b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:13:17.256120 4772 raft_consensus.cc:482] T 6a5f99276dad456ea80e3c9f9618059a P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:17.256306 4772 raft_consensus.cc:504] T 6a5f99276dad456ea80e3c9f9618059a P aad81047009a4ffcbc450b60030583b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } }
I20240501 08:13:17.256847 4772 leader_election.cc:288] T 6a5f99276dad456ea80e3c9f9618059a P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:13:17.261909 28015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb7dae019c7e4842b0daa35ee4b40367" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:13:17.266547 30472 leader_election.cc:302] T bb7dae019c7e4842b0daa35ee4b40367 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31811708c6774037b82df44024b229b6; no voters: 6331f77dae9d452184e8cd0cbf5f1b43, aad81047009a4ffcbc450b60030583b5
W20240501 08:13:17.267195 27958 consensus_peers.cc:485] T 19f899c9c362457a9007aa19ec685fd0 P aad81047009a4ffcbc450b60030583b5 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:17.267529 4734 raft_consensus.cc:2726] T bb7dae019c7e4842b0daa35ee4b40367 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:13:17.269026 27958 consensus_peers.cc:485] T 87bd346903dc4c2c85ecaf362fe57799 P aad81047009a4ffcbc450b60030583b5 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:17.265863 28010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ac9368b897246ac8b2f6b2f096ff3ea" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "aad81047009a4ffcbc450b60030583b5" is_pre_election: true
I20240501 08:13:17.270153 30607 leader_election.cc:302] T 0ac9368b897246ac8b2f6b2f096ff3ea P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a022e1a501f4d44a0468054d6c3de47; no voters: 6331f77dae9d452184e8cd0cbf5f1b43, aad81047009a4ffcbc450b60030583b5
W20240501 08:13:17.270128 27958 leader_election.cc:334] T 6a5f99276dad456ea80e3c9f9618059a P aad81047009a4ffcbc450b60030583b5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
I20240501 08:13:17.270502 4776 raft_consensus.cc:2726] T 0ac9368b897246ac8b2f6b2f096ff3ea P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:13:17.272291 4777 raft_consensus.cc:482] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:17.272464 4777 raft_consensus.cc:504] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:13:17.273244 4777 leader_election.cc:288] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:13:17.273734 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 27940
I20240501 08:13:17.275840 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e94895331d6b474484d987d42f13fd88" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:13:17.276106 30803 raft_consensus.cc:2454] T e94895331d6b474484d987d42f13fd88 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 1.
W20240501 08:13:17.277268 30607 leader_election.cc:334] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
I20240501 08:13:17.278651 30606 leader_election.cc:302] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 9a022e1a501f4d44a0468054d6c3de47; no voters: 6331f77dae9d452184e8cd0cbf5f1b43
I20240501 08:13:17.278946 4777 raft_consensus.cc:2781] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:13:17.279040 4777 raft_consensus.cc:482] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:17.279112 4777 raft_consensus.cc:3037] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:17.281595 4777 raft_consensus.cc:504] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:13:17.282114 4777 leader_election.cc:288] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 election: Requested vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:13:17.282604 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e94895331d6b474484d987d42f13fd88" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:13:17.282788 30803 raft_consensus.cc:3037] T e94895331d6b474484d987d42f13fd88 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Advancing to term 2
W20240501 08:13:17.283811 30607 leader_election.cc:334] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
I20240501 08:13:17.285543 4734 raft_consensus.cc:482] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:17.285609 30803 raft_consensus.cc:2454] T e94895331d6b474484d987d42f13fd88 P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 2.
I20240501 08:13:17.285729 4734 raft_consensus.cc:504] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:13:17.286047 30606 leader_election.cc:302] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 9a022e1a501f4d44a0468054d6c3de47; no voters: 6331f77dae9d452184e8cd0cbf5f1b43
I20240501 08:13:17.286315 4777 raft_consensus.cc:2781] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:17.286482 4777 raft_consensus.cc:686] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47 [term 2 LEADER]: Becoming Leader. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Running, Role: LEADER
I20240501 08:13:17.286617 4734 leader_election.cc:288] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:13:17.286849 4777 consensus_queue.cc:237] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:13:17.287318 28149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63215605ade41a5bc8d97a35ce33a31" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:13:17.287523 28149 raft_consensus.cc:2454] T f63215605ade41a5bc8d97a35ce33a31 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 2.
I20240501 08:13:17.288400 30470 leader_election.cc:302] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:13:17.288722 4734 raft_consensus.cc:2781] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 08:13:17.288810 4734 raft_consensus.cc:482] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:17.288883 4734 raft_consensus.cc:3037] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Advancing to term 3
W20240501 08:13:17.291414 30472 leader_election.cc:334] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
I20240501 08:13:17.293514 4773 raft_consensus.cc:482] T 948f6afb45594870bfe44b45d303f88f P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f1525cbc82a24d13b65f136695c903af)
I20240501 08:13:17.293660 4773 raft_consensus.cc:504] T 948f6afb45594870bfe44b45d303f88f P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:13:17.294270 4773 leader_election.cc:288] T 948f6afb45594870bfe44b45d303f88f P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:13:17.295085 4734 raft_consensus.cc:504] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
W20240501 08:13:17.296039 30473 leader_election.cc:334] T 948f6afb45594870bfe44b45d303f88f P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
I20240501 08:13:17.296141 4734 leader_election.cc:288] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 3 election: Requested vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:13:17.293910 27612 catalog_manager.cc:5484] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: term changed from 1 to 2, leader changed from 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) to 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134). New cstate: current_term: 2 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:13:17.296423 28149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63215605ade41a5bc8d97a35ce33a31" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:13:17.296607 28149 raft_consensus.cc:3037] T f63215605ade41a5bc8d97a35ce33a31 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:17.296743 30934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "948f6afb45594870bfe44b45d303f88f" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:13:17.298971 30472 leader_election.cc:302] T 948f6afb45594870bfe44b45d303f88f P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31811708c6774037b82df44024b229b6; no voters: a29cd590999b4f32849ff5689cd53c11, f1525cbc82a24d13b65f136695c903af
I20240501 08:13:17.301424 28149 raft_consensus.cc:2454] T f63215605ade41a5bc8d97a35ce33a31 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 3.
W20240501 08:13:17.301870 30472 leader_election.cc:334] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
I20240501 08:13:17.302129 30470 leader_election.cc:302] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31811708c6774037b82df44024b229b6, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters: 6331f77dae9d452184e8cd0cbf5f1b43
I20240501 08:13:17.302433 4774 raft_consensus.cc:2726] T 948f6afb45594870bfe44b45d303f88f P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:13:17.302515 4734 raft_consensus.cc:2781] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:13:17.302673 4734 raft_consensus.cc:686] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [term 3 LEADER]: Becoming Leader. State: Replica: 31811708c6774037b82df44024b229b6, State: Running, Role: LEADER
I20240501 08:13:17.303423 4734 consensus_queue.cc:237] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:13:17.320665 27612 catalog_manager.cc:5484] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 reported cstate change: term changed from 2 to 3, leader changed from 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) to 31811708c6774037b82df44024b229b6 (127.21.84.133). New cstate: current_term: 3 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:17.335911 4776 raft_consensus.cc:482] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:17.336072 4776 raft_consensus.cc:504] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:13:17.336692 4776 leader_election.cc:288] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:13:17.337205 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66b8b9c025b941a1a1b6e681fa0bbe90" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:13:17.337400 30803 raft_consensus.cc:2454] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 1.
I20240501 08:13:17.337750 30606 leader_election.cc:302] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:13:17.338016 4776 raft_consensus.cc:2781] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:13:17.338083 4776 raft_consensus.cc:482] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:17.338156 4776 raft_consensus.cc:3037] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Advancing to term 2
W20240501 08:13:17.338439 30607 leader_election.cc:334] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
I20240501 08:13:17.340327 4776 raft_consensus.cc:504] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 9a022e1a501f4d44a0468054d6c3de47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:13:17.341406 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66b8b9c025b941a1a1b6e681fa0bbe90" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:13:17.341591 30803 raft_consensus.cc:3037] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:17.342060 4740 raft_consensus.cc:482] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
W20240501 08:13:17.342293 30607 leader_election.cc:334] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
I20240501 08:13:17.342211 4740 raft_consensus.cc:504] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:13:17.343022 4740 leader_election.cc:288] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:13:17.343833 30803 raft_consensus.cc:2454] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 2.
I20240501 08:13:17.343892 28149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "728616fd583a4afea00e8fe52880e55b" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:13:17.344115 28149 raft_consensus.cc:2454] T 728616fd583a4afea00e8fe52880e55b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 2.
I20240501 08:13:17.344501 30606 leader_election.cc:302] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 9a022e1a501f4d44a0468054d6c3de47; no voters: 6331f77dae9d452184e8cd0cbf5f1b43
I20240501 08:13:17.344539 30737 leader_election.cc:302] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
I20240501 08:13:17.344780 4799 raft_consensus.cc:2781] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 9a022e1a501f4d44a0468054d6c3de47 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:17.344823 4740 raft_consensus.cc:2781] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20240501 08:13:17.344873 30739 leader_election.cc:334] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
I20240501 08:13:17.344919 4799 raft_consensus.cc:686] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 9a022e1a501f4d44a0468054d6c3de47 [term 2 LEADER]: Becoming Leader. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Running, Role: LEADER
I20240501 08:13:17.344893 4740 raft_consensus.cc:482] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:17.345039 4740 raft_consensus.cc:3037] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:17.345242 4799 consensus_queue.cc:237] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:13:17.346181 4776 leader_election.cc:288] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 election: Requested vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:13:17.348095 4740 raft_consensus.cc:504] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
W20240501 08:13:17.348953 30605 connection.cc:535] server connection from 127.21.84.131:44127 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:17.349282 30872 connection.cc:535] server connection from 127.21.84.131:60955 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:17.350029 30607 connection.cc:535] client connection to 127.21.84.131:36305 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:17.350023 30472 connection.cc:535] client connection to 127.21.84.131:36305 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:13:17.350140 28149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "728616fd583a4afea00e8fe52880e55b" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6"
I20240501 08:13:17.350322 28149 raft_consensus.cc:3037] T 728616fd583a4afea00e8fe52880e55b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Advancing to term 3
W20240501 08:13:17.350481 28087 connection.cc:535] client connection to 127.21.84.131:36305 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:17.352828 4768 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 08:13:17.351169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.84.131:36305 (local address 127.21.84.132:60053)
0501 08:13:17.351378 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0501 08:13:17.351397 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0501 08:13:17.351513 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 08:13:17.351874 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":117}
W20240501 08:13:17.353042 28087 consensus_peers.cc:485] T 79bd880115e1444594881c95c55016d2 P 91d11eae5ef6432ca86eb8ed446b6ea6 -> Peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Couldn't send request to peer aad81047009a4ffcbc450b60030583b5. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:17.353102 28149 raft_consensus.cc:2454] T 728616fd583a4afea00e8fe52880e55b P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 3.
I20240501 08:13:17.353734 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 28070
W20240501 08:13:17.354027 4788 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 08:13:17.351039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.84.131:36305 (local address 127.21.84.133:49599)
0501 08:13:17.353674 (+ 2635us) negotiation.cc:107] Waiting for socket to connect
0501 08:13:17.353802 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":2516}
I20240501 08:13:17.355082 30737 leader_election.cc:302] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 91d11eae5ef6432ca86eb8ed446b6ea6; no voters:
W20240501 08:13:17.355422 30739 leader_election.cc:334] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
I20240501 08:13:17.355460 4778 raft_consensus.cc:2781] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:13:17.355700 4778 raft_consensus.cc:686] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5 [term 3 LEADER]: Becoming Leader. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Running, Role: LEADER
I20240501 08:13:17.356016 4740 leader_election.cc:288] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 3 election: Requested vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
W20240501 08:13:17.356001 30472 consensus_peers.cc:485] T 7affd631d1e444eaaf8a4c51267a204f P 31811708c6774037b82df44024b229b6 -> Peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Couldn't send request to peer aad81047009a4ffcbc450b60030583b5. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:17.355988 4778 consensus_queue.cc:237] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
W20240501 08:13:17.359544 30607 consensus_peers.cc:485] T 4973818ff02b4b7eba922b7fe9193ca6 P 9a022e1a501f4d44a0468054d6c3de47 -> Peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Couldn't send request to peer aad81047009a4ffcbc450b60030583b5. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:17.359745 4776 raft_consensus.cc:482] T 103e1394bfe34a86ba1d9487d75fd512 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:17.359895 4776 raft_consensus.cc:504] T 103e1394bfe34a86ba1d9487d75fd512 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
W20240501 08:13:17.363191 30607 leader_election.cc:334] T 103e1394bfe34a86ba1d9487d75fd512 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
W20240501 08:13:17.367076 30608 leader_election.cc:334] T 103e1394bfe34a86ba1d9487d75fd512 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
I20240501 08:13:17.367218 30608 leader_election.cc:302] T 103e1394bfe34a86ba1d9487d75fd512 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a022e1a501f4d44a0468054d6c3de47; no voters: 6331f77dae9d452184e8cd0cbf5f1b43, f1525cbc82a24d13b65f136695c903af
I20240501 08:13:17.367539 4776 leader_election.cc:288] T 103e1394bfe34a86ba1d9487d75fd512 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:13:17.367699 4799 raft_consensus.cc:2726] T 103e1394bfe34a86ba1d9487d75fd512 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:13:17.370678 4734 raft_consensus.cc:482] T 019d8108d6d84d608fd8d5237f3f20ff P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:17.370837 4734 raft_consensus.cc:504] T 019d8108d6d84d608fd8d5237f3f20ff P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:13:17.371086 27612 catalog_manager.cc:5484] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: term changed from 1 to 2, leader changed from 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) to 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134). New cstate: current_term: 2 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:17.371565 4734 leader_election.cc:288] T 019d8108d6d84d608fd8d5237f3f20ff P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:13:17.371338 27613 catalog_manager.cc:5484] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5 reported cstate change: term changed from 2 to 3, leader changed from 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) to 64998323a1f4482d8704bf20594d13a5 (127.21.84.135). New cstate: current_term: 3 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:13:17.372498 30934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019d8108d6d84d608fd8d5237f3f20ff" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:13:17.372689 4795 raft_consensus.cc:482] T 4fd595c79cdc43f9a8afadb6b548bf64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f1525cbc82a24d13b65f136695c903af)
I20240501 08:13:17.372828 4795 raft_consensus.cc:504] T 4fd595c79cdc43f9a8afadb6b548bf64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:13:17.373607 4795 leader_election.cc:288] T 4fd595c79cdc43f9a8afadb6b548bf64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:13:17.374222 30668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fd595c79cdc43f9a8afadb6b548bf64" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
W20240501 08:13:17.375051 30472 leader_election.cc:334] T 019d8108d6d84d608fd8d5237f3f20ff P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
I20240501 08:13:17.375154 30472 leader_election.cc:302] T 019d8108d6d84d608fd8d5237f3f20ff P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31811708c6774037b82df44024b229b6; no voters: 6331f77dae9d452184e8cd0cbf5f1b43, a29cd590999b4f32849ff5689cd53c11
W20240501 08:13:17.375186 30872 consensus_peers.cc:485] T a3f0322a212a459b925a0f4e2dd2457e P a29cd590999b4f32849ff5689cd53c11 -> Peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Couldn't send request to peer aad81047009a4ffcbc450b60030583b5. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:17.375399 4734 raft_consensus.cc:2726] T 019d8108d6d84d608fd8d5237f3f20ff P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:13:17.377724 28088 leader_election.cc:334] T 4fd595c79cdc43f9a8afadb6b548bf64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
I20240501 08:13:17.377825 28088 leader_election.cc:302] T 4fd595c79cdc43f9a8afadb6b548bf64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6; no voters: 9a022e1a501f4d44a0468054d6c3de47, f1525cbc82a24d13b65f136695c903af
W20240501 08:13:17.378691 30472 consensus_peers.cc:485] T 4f8c0a8e86f04f32b79743f819948544 P 31811708c6774037b82df44024b229b6 -> Peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Couldn't send request to peer aad81047009a4ffcbc450b60030583b5. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:17.378691 4795 raft_consensus.cc:2726] T 4fd595c79cdc43f9a8afadb6b548bf64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:13:17.382473 28087 consensus_peers.cc:485] T a0f992f08fb841a5b18b14f5ad0b9c58 P 91d11eae5ef6432ca86eb8ed446b6ea6 -> Peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Couldn't send request to peer aad81047009a4ffcbc450b60030583b5. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:17.382695 28087 consensus_peers.cc:485] T 8eb9763a1573455287b7137fe7fecf62 P 91d11eae5ef6432ca86eb8ed446b6ea6 -> Peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Couldn't send request to peer aad81047009a4ffcbc450b60030583b5. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:17.386164 28192 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:13:17.388438 4734 raft_consensus.cc:482] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:17.388595 4734 raft_consensus.cc:504] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:13:17.389221 4734 leader_election.cc:288] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:13:17.389784 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f40a7ef6954b34a48d8c5aeeaebc0a" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:13:17.390010 30803 raft_consensus.cc:2454] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 2.
W20240501 08:13:17.390291 30472 leader_election.cc:334] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
I20240501 08:13:17.390467 30471 leader_election.cc:302] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31811708c6774037b82df44024b229b6, 64998323a1f4482d8704bf20594d13a5; no voters: 6331f77dae9d452184e8cd0cbf5f1b43
I20240501 08:13:17.390707 4734 raft_consensus.cc:2781] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 08:13:17.390766 4734 raft_consensus.cc:482] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:17.390810 4734 raft_consensus.cc:3037] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:17.392637 4734 raft_consensus.cc:504] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:13:17.393199 4734 leader_election.cc:288] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 3 election: Requested vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:13:17.393646 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f40a7ef6954b34a48d8c5aeeaebc0a" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5"
I20240501 08:13:17.393819 30803 raft_consensus.cc:3037] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Advancing to term 3
W20240501 08:13:17.394583 30472 leader_election.cc:334] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
I20240501 08:13:17.395797 30803 raft_consensus.cc:2454] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 64998323a1f4482d8704bf20594d13a5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 3.
I20240501 08:13:17.396183 30471 leader_election.cc:302] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31811708c6774037b82df44024b229b6, 64998323a1f4482d8704bf20594d13a5; no voters: 6331f77dae9d452184e8cd0cbf5f1b43
I20240501 08:13:17.396495 4734 raft_consensus.cc:2781] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:13:17.396610 4734 raft_consensus.cc:686] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6 [term 3 LEADER]: Becoming Leader. State: Replica: 31811708c6774037b82df44024b229b6, State: Running, Role: LEADER
I20240501 08:13:17.396894 4734 consensus_queue.cc:237] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:13:17.402592 27613 catalog_manager.cc:5484] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6 reported cstate change: term changed from 2 to 3, leader changed from 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) to 31811708c6774037b82df44024b229b6 (127.21.84.133). New cstate: current_term: 3 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:17.408864 4734 raft_consensus.cc:482] T 802538381de843ab8d326014a0b4ec15 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:17.409123 4734 raft_consensus.cc:504] T 802538381de843ab8d326014a0b4ec15 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } }
I20240501 08:13:17.409291 4778 raft_consensus.cc:482] T 9b5b0b1e46d14d78806859541e0ab3b6 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:17.409628 4778 raft_consensus.cc:504] T 9b5b0b1e46d14d78806859541e0ab3b6 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:13:17.410658 30668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "802538381de843ab8d326014a0b4ec15" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:13:17.411003 4734 leader_election.cc:288] T 802538381de843ab8d326014a0b4ec15 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
W20240501 08:13:17.412194 30472 leader_election.cc:334] T 802538381de843ab8d326014a0b4ec15 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
W20240501 08:13:17.412436 30472 consensus_peers.cc:485] T b16391636d5f4ec48c9e4f18ed40badd P 31811708c6774037b82df44024b229b6 -> Peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Couldn't send request to peer aad81047009a4ffcbc450b60030583b5. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:17.412425 30470 leader_election.cc:302] T 802538381de843ab8d326014a0b4ec15 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31811708c6774037b82df44024b229b6; no voters: 6331f77dae9d452184e8cd0cbf5f1b43, 9a022e1a501f4d44a0468054d6c3de47
I20240501 08:13:17.412716 4734 raft_consensus.cc:2726] T 802538381de843ab8d326014a0b4ec15 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:13:17.413043 4778 leader_election.cc:288] T 9b5b0b1e46d14d78806859541e0ab3b6 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:13:17.413379 30534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b5b0b1e46d14d78806859541e0ab3b6" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:13:17.413540 30534 raft_consensus.cc:2454] T 9b5b0b1e46d14d78806859541e0ab3b6 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:13:17.414637 30740 leader_election.cc:302] T 9b5b0b1e46d14d78806859541e0ab3b6 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 64998323a1f4482d8704bf20594d13a5; no voters:
I20240501 08:13:17.414955 4778 raft_consensus.cc:2781] T 9b5b0b1e46d14d78806859541e0ab3b6 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:13:17.415035 4778 raft_consensus.cc:482] T 9b5b0b1e46d14d78806859541e0ab3b6 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
W20240501 08:13:17.415091 30739 leader_election.cc:334] T 9b5b0b1e46d14d78806859541e0ab3b6 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
I20240501 08:13:17.415099 4778 raft_consensus.cc:3037] T 9b5b0b1e46d14d78806859541e0ab3b6 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Advancing to term 2
W20240501 08:13:17.415619 30872 consensus_peers.cc:485] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 -> Peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Couldn't send request to peer aad81047009a4ffcbc450b60030583b5. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:17.418442 4778 raft_consensus.cc:504] T 9b5b0b1e46d14d78806859541e0ab3b6 P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:13:17.419495 30534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b5b0b1e46d14d78806859541e0ab3b6" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:13:17.419665 30534 raft_consensus.cc:3037] T 9b5b0b1e46d14d78806859541e0ab3b6 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Advancing to term 2
W20240501 08:13:17.420958 30739 leader_election.cc:334] T 9b5b0b1e46d14d78806859541e0ab3b6 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
I20240501 08:13:17.421169 4778 leader_election.cc:288] T 9b5b0b1e46d14d78806859541e0ab3b6 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 election: Requested vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:13:17.422319 30534 raft_consensus.cc:2454] T 9b5b0b1e46d14d78806859541e0ab3b6 P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 2.
I20240501 08:13:17.423040 30740 leader_election.cc:302] T 9b5b0b1e46d14d78806859541e0ab3b6 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31811708c6774037b82df44024b229b6, 64998323a1f4482d8704bf20594d13a5; no voters: 6331f77dae9d452184e8cd0cbf5f1b43
I20240501 08:13:17.423498 4778 raft_consensus.cc:2781] T 9b5b0b1e46d14d78806859541e0ab3b6 P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:17.423661 4778 raft_consensus.cc:686] T 9b5b0b1e46d14d78806859541e0ab3b6 P 64998323a1f4482d8704bf20594d13a5 [term 2 LEADER]: Becoming Leader. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Running, Role: LEADER
I20240501 08:13:17.423931 4778 consensus_queue.cc:237] T 9b5b0b1e46d14d78806859541e0ab3b6 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:13:17.438232 4734 raft_consensus.cc:482] T 0e0f87fa9df24b43bedf063a9063bcc5 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:17.438448 4734 raft_consensus.cc:504] T 0e0f87fa9df24b43bedf063a9063bcc5 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } }
I20240501 08:13:17.438953 4776 raft_consensus.cc:482] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:17.439085 4734 leader_election.cc:288] T 0e0f87fa9df24b43bedf063a9063bcc5 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:13:17.439102 4776 raft_consensus.cc:504] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } }
I20240501 08:13:17.439870 4776 leader_election.cc:288] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:13:17.440459 30534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "802538381de843ab8d326014a0b4ec15" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:13:17.440630 30534 raft_consensus.cc:2454] T 802538381de843ab8d326014a0b4ec15 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 1.
W20240501 08:13:17.441267 30739 consensus_peers.cc:485] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 -> Peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Couldn't send request to peer aad81047009a4ffcbc450b60030583b5. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:17.441385 30608 leader_election.cc:302] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, 9a022e1a501f4d44a0468054d6c3de47; no voters:
I20240501 08:13:17.442759 4799 raft_consensus.cc:2781] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:13:17.442835 4799 raft_consensus.cc:482] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:17.442888 4799 raft_consensus.cc:3037] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Advancing to term 2
W20240501 08:13:17.444097 30472 leader_election.cc:334] T 0e0f87fa9df24b43bedf063a9063bcc5 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
W20240501 08:13:17.444193 30473 leader_election.cc:334] T 0e0f87fa9df24b43bedf063a9063bcc5 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
I20240501 08:13:17.444290 30473 leader_election.cc:302] T 0e0f87fa9df24b43bedf063a9063bcc5 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31811708c6774037b82df44024b229b6; no voters: 6331f77dae9d452184e8cd0cbf5f1b43, f1525cbc82a24d13b65f136695c903af
I20240501 08:13:17.444626 4734 raft_consensus.cc:2726] T 0e0f87fa9df24b43bedf063a9063bcc5 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:13:17.444842 30607 leader_election.cc:334] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
I20240501 08:13:17.445786 4799 raft_consensus.cc:504] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } }
I20240501 08:13:17.446504 4799 leader_election.cc:288] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 election: Requested vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:13:17.446897 30534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "802538381de843ab8d326014a0b4ec15" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:13:17.447077 30534 raft_consensus.cc:3037] T 802538381de843ab8d326014a0b4ec15 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:17.447995 4727 raft_consensus.cc:482] T f3b6691cbe8740d2978c20209c78f3ff P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:17.448172 4727 raft_consensus.cc:504] T f3b6691cbe8740d2978c20209c78f3ff P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:13:17.448874 4734 raft_consensus.cc:482] T 3aa09c722530431384e5e7ee9759230c P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:17.448920 4727 leader_election.cc:288] T f3b6691cbe8740d2978c20209c78f3ff P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:13:17.449002 4734 raft_consensus.cc:504] T 3aa09c722530431384e5e7ee9759230c P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:13:17.449625 4734 leader_election.cc:288] T 3aa09c722530431384e5e7ee9759230c P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
W20240501 08:13:17.452072 30472 leader_election.cc:334] T 3aa09c722530431384e5e7ee9759230c P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
I20240501 08:13:17.453254 4734 raft_consensus.cc:482] T 7f7211c8c0e041bf89930aa50a4438ea P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f1525cbc82a24d13b65f136695c903af)
I20240501 08:13:17.453382 4734 raft_consensus.cc:504] T 7f7211c8c0e041bf89930aa50a4438ea P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:13:17.454013 4734 leader_election.cc:288] T 7f7211c8c0e041bf89930aa50a4438ea P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
W20240501 08:13:17.454877 30607 leader_election.cc:334] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
I20240501 08:13:17.454838 30934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f7211c8c0e041bf89930aa50a4438ea" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:13:17.458139 27613 catalog_manager.cc:5484] T 9b5b0b1e46d14d78806859541e0ab3b6 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: term changed from 1 to 2, leader changed from 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) to 64998323a1f4482d8704bf20594d13a5 (127.21.84.135). New cstate: current_term: 2 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } }
W20240501 08:13:17.460482 30872 leader_election.cc:334] T f3b6691cbe8740d2978c20209c78f3ff P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
W20240501 08:13:17.461884 30473 leader_election.cc:334] T 7f7211c8c0e041bf89930aa50a4438ea P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
I20240501 08:13:17.462028 30473 leader_election.cc:302] T 7f7211c8c0e041bf89930aa50a4438ea P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31811708c6774037b82df44024b229b6; no voters: a29cd590999b4f32849ff5689cd53c11, f1525cbc82a24d13b65f136695c903af
I20240501 08:13:17.462431 4734 raft_consensus.cc:2726] T 7f7211c8c0e041bf89930aa50a4438ea P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:13:17.475128 30534 raft_consensus.cc:2454] T 802538381de843ab8d326014a0b4ec15 P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 2.
I20240501 08:13:17.475790 30608 leader_election.cc:302] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31811708c6774037b82df44024b229b6, 9a022e1a501f4d44a0468054d6c3de47; no voters: 6331f77dae9d452184e8cd0cbf5f1b43
I20240501 08:13:17.476272 4799 raft_consensus.cc:2781] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:17.476390 4799 raft_consensus.cc:686] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47 [term 2 LEADER]: Becoming Leader. State: Replica: 9a022e1a501f4d44a0468054d6c3de47, State: Running, Role: LEADER
I20240501 08:13:17.476650 4799 consensus_queue.cc:237] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } }
I20240501 08:13:17.491083 27613 catalog_manager.cc:5484] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47 reported cstate change: term changed from 1 to 2, leader changed from 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) to 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134). New cstate: current_term: 2 leader_uuid: "9a022e1a501f4d44a0468054d6c3de47" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 08:13:17.497805 30739 consensus_peers.cc:485] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 -> Peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Couldn't send request to peer aad81047009a4ffcbc450b60030583b5. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:17.512619 3963 raft_consensus.cc:482] T 98c8e3a6f66a42b08507668f7c3a8af1 P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f1525cbc82a24d13b65f136695c903af)
I20240501 08:13:17.512815 3963 raft_consensus.cc:504] T 98c8e3a6f66a42b08507668f7c3a8af1 P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:13:17.513582 3963 leader_election.cc:288] T 98c8e3a6f66a42b08507668f7c3a8af1 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:13:17.514204 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c8e3a6f66a42b08507668f7c3a8af1" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
W20240501 08:13:17.515085 30873 leader_election.cc:334] T 98c8e3a6f66a42b08507668f7c3a8af1 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
I20240501 08:13:17.515365 30871 leader_election.cc:302] T 98c8e3a6f66a42b08507668f7c3a8af1 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a29cd590999b4f32849ff5689cd53c11; no voters: 64998323a1f4482d8704bf20594d13a5, f1525cbc82a24d13b65f136695c903af
I20240501 08:13:17.515632 3963 raft_consensus.cc:2726] T 98c8e3a6f66a42b08507668f7c3a8af1 P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:13:17.533102 30739 consensus_peers.cc:485] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 -> Peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Couldn't send request to peer aad81047009a4ffcbc450b60030583b5. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:17.569700 3963 raft_consensus.cc:482] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f1525cbc82a24d13b65f136695c903af)
I20240501 08:13:17.569904 3963 raft_consensus.cc:504] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:13:17.570693 3963 leader_election.cc:288] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:13:17.571528 30534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05eac5f1b08945f8b7b8c78d0f4d3d3a" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
W20240501 08:13:17.572057 30873 leader_election.cc:334] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
I20240501 08:13:17.572162 30873 leader_election.cc:302] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a29cd590999b4f32849ff5689cd53c11; no voters: 31811708c6774037b82df44024b229b6, f1525cbc82a24d13b65f136695c903af
I20240501 08:13:17.572402 3963 raft_consensus.cc:2726] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:13:17.593684 4734 raft_consensus.cc:482] T 9eecbc3985624f3889d8e554cd585b11 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f1525cbc82a24d13b65f136695c903af)
I20240501 08:13:17.593881 4734 raft_consensus.cc:504] T 9eecbc3985624f3889d8e554cd585b11 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:13:17.594482 4734 leader_election.cc:288] T 9eecbc3985624f3889d8e554cd585b11 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
W20240501 08:13:17.595862 30472 leader_election.cc:334] T 9eecbc3985624f3889d8e554cd585b11 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: connect: Connection refused (error 111)
W20240501 08:13:17.595984 30473 leader_election.cc:334] T 9eecbc3985624f3889d8e554cd585b11 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
I20240501 08:13:17.596069 30473 leader_election.cc:302] T 9eecbc3985624f3889d8e554cd585b11 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31811708c6774037b82df44024b229b6; no voters: aad81047009a4ffcbc450b60030583b5, f1525cbc82a24d13b65f136695c903af
I20240501 08:13:17.596267 4734 raft_consensus.cc:2726] T 9eecbc3985624f3889d8e554cd585b11 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:13:17.603111 4799 raft_consensus.cc:482] T 4fd595c79cdc43f9a8afadb6b548bf64 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f1525cbc82a24d13b65f136695c903af)
I20240501 08:13:17.603283 4799 raft_consensus.cc:504] T 4fd595c79cdc43f9a8afadb6b548bf64 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:13:17.603989 4799 leader_election.cc:288] T 4fd595c79cdc43f9a8afadb6b548bf64 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
W20240501 08:13:17.605109 30608 leader_election.cc:334] T 4fd595c79cdc43f9a8afadb6b548bf64 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
W20240501 08:13:17.613991 30739 consensus_peers.cc:485] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 -> Peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Couldn't send request to peer aad81047009a4ffcbc450b60030583b5. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:17.615411 30472 consensus_peers.cc:485] T 1448b8f2d5844ad7aec9bcd3874c5c01 P 31811708c6774037b82df44024b229b6 -> Peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Couldn't send request to peer aad81047009a4ffcbc450b60030583b5. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:17.615535 30472 consensus_peers.cc:485] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 -> Peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Couldn't send request to peer aad81047009a4ffcbc450b60030583b5. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:17.632961 3963 raft_consensus.cc:482] T 2ce43dc81a1b42c3a63124490d8e60cd P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f1525cbc82a24d13b65f136695c903af)
I20240501 08:13:17.633168 3963 raft_consensus.cc:504] T 2ce43dc81a1b42c3a63124490d8e60cd P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:13:17.633877 3963 leader_election.cc:288] T 2ce43dc81a1b42c3a63124490d8e60cd P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
W20240501 08:13:17.635371 30873 leader_election.cc:334] T 2ce43dc81a1b42c3a63124490d8e60cd P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
W20240501 08:13:17.635509 30872 leader_election.cc:334] T 2ce43dc81a1b42c3a63124490d8e60cd P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: connect: Connection refused (error 111)
I20240501 08:13:17.635623 30872 leader_election.cc:302] T 2ce43dc81a1b42c3a63124490d8e60cd P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a29cd590999b4f32849ff5689cd53c11; no voters: aad81047009a4ffcbc450b60030583b5, f1525cbc82a24d13b65f136695c903af
I20240501 08:13:17.635850 3963 raft_consensus.cc:2726] T 2ce43dc81a1b42c3a63124490d8e60cd P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:13:17.643270 3963 raft_consensus.cc:482] T 24825c85238b4d618c65d00107135420 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f1525cbc82a24d13b65f136695c903af)
I20240501 08:13:17.643471 3963 raft_consensus.cc:504] T 24825c85238b4d618c65d00107135420 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:13:17.644033 3963 leader_election.cc:288] T 24825c85238b4d618c65d00107135420 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
W20240501 08:13:17.645223 30873 leader_election.cc:334] T 24825c85238b4d618c65d00107135420 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
W20240501 08:13:17.645373 30872 leader_election.cc:334] T 24825c85238b4d618c65d00107135420 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
I20240501 08:13:17.645469 30872 leader_election.cc:302] T 24825c85238b4d618c65d00107135420 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a29cd590999b4f32849ff5689cd53c11; no voters: 6331f77dae9d452184e8cd0cbf5f1b43, f1525cbc82a24d13b65f136695c903af
I20240501 08:13:17.645680 3963 raft_consensus.cc:2726] T 24825c85238b4d618c65d00107135420 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:13:17.655318 3963 raft_consensus.cc:482] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f1525cbc82a24d13b65f136695c903af)
I20240501 08:13:17.655506 3963 raft_consensus.cc:504] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:13:17.656082 3963 leader_election.cc:288] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:13:17.656903 30534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f7211c8c0e041bf89930aa50a4438ea" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:13:17.657119 30534 raft_consensus.cc:2454] T 7f7211c8c0e041bf89930aa50a4438ea P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:13:17.657727 30873 leader_election.cc:302] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters:
W20240501 08:13:17.657981 30873 leader_election.cc:334] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
I20240501 08:13:17.658324 3963 raft_consensus.cc:2781] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:13:17.658398 3963 raft_consensus.cc:482] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f1525cbc82a24d13b65f136695c903af)
I20240501 08:13:17.658461 3963 raft_consensus.cc:3037] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:17.661382 3963 raft_consensus.cc:504] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:13:17.662096 3963 leader_election.cc:288] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 election: Requested vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:13:17.662988 30534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f7211c8c0e041bf89930aa50a4438ea" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:13:17.663182 30534 raft_consensus.cc:3037] T 7f7211c8c0e041bf89930aa50a4438ea P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Advancing to term 2
W20240501 08:13:17.663602 30873 leader_election.cc:334] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
I20240501 08:13:17.665974 30534 raft_consensus.cc:2454] T 7f7211c8c0e041bf89930aa50a4438ea P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 2.
I20240501 08:13:17.666360 30873 leader_election.cc:302] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters: f1525cbc82a24d13b65f136695c903af
I20240501 08:13:17.666635 3963 raft_consensus.cc:2781] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:17.666774 3963 raft_consensus.cc:686] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11 [term 2 LEADER]: Becoming Leader. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:13:17.667101 3963 consensus_queue.cc:237] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:13:17.671744 27613 catalog_manager.cc:5484] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11 reported cstate change: term changed from 1 to 2, leader changed from f1525cbc82a24d13b65f136695c903af (127.21.84.130) to a29cd590999b4f32849ff5689cd53c11 (127.21.84.136). New cstate: current_term: 2 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 08:13:17.686612 4779 raft_consensus.cc:482] T a2981c0863bc41729605f8b076b6a6cb P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f1525cbc82a24d13b65f136695c903af)
I20240501 08:13:17.686811 4779 raft_consensus.cc:504] T a2981c0863bc41729605f8b076b6a6cb P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:13:17.687502 4779 leader_election.cc:288] T a2981c0863bc41729605f8b076b6a6cb P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:13:17.687713 4734 raft_consensus.cc:482] T d6bb70263134414b9945a87087b55a2c P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f1525cbc82a24d13b65f136695c903af)
I20240501 08:13:17.687882 4734 raft_consensus.cc:504] T d6bb70263134414b9945a87087b55a2c P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } }
I20240501 08:13:17.688462 4734 leader_election.cc:288] T d6bb70263134414b9945a87087b55a2c P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
W20240501 08:13:17.689922 30873 leader_election.cc:334] T a2981c0863bc41729605f8b076b6a6cb P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
W20240501 08:13:17.690263 30473 leader_election.cc:334] T d6bb70263134414b9945a87087b55a2c P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
W20240501 08:13:17.690441 30472 leader_election.cc:334] T d6bb70263134414b9945a87087b55a2c P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: connect: Connection refused (error 111)
I20240501 08:13:17.690534 30472 leader_election.cc:302] T d6bb70263134414b9945a87087b55a2c P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31811708c6774037b82df44024b229b6; no voters: aad81047009a4ffcbc450b60030583b5, f1525cbc82a24d13b65f136695c903af
I20240501 08:13:17.690747 4734 raft_consensus.cc:2726] T d6bb70263134414b9945a87087b55a2c P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:13:17.707813 30872 consensus_peers.cc:485] T c832fea33e4744868b69e6c29e4aea9f P a29cd590999b4f32849ff5689cd53c11 -> Peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Couldn't send request to peer aad81047009a4ffcbc450b60030583b5. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:17.715010 3963 raft_consensus.cc:482] T 0e06cf862c954e8fb1ca087ec5184497 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f1525cbc82a24d13b65f136695c903af)
I20240501 08:13:17.715173 3963 raft_consensus.cc:504] T 0e06cf862c954e8fb1ca087ec5184497 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:13:17.715780 3963 leader_election.cc:288] T 0e06cf862c954e8fb1ca087ec5184497 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
W20240501 08:13:17.717168 30873 leader_election.cc:334] T 0e06cf862c954e8fb1ca087ec5184497 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
W20240501 08:13:17.717698 30872 leader_election.cc:334] T 0e06cf862c954e8fb1ca087ec5184497 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: connect: Connection refused (error 111)
I20240501 08:13:17.717803 30872 leader_election.cc:302] T 0e06cf862c954e8fb1ca087ec5184497 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a29cd590999b4f32849ff5689cd53c11; no voters: aad81047009a4ffcbc450b60030583b5, f1525cbc82a24d13b65f136695c903af
I20240501 08:13:17.718392 3963 raft_consensus.cc:2726] T 0e06cf862c954e8fb1ca087ec5184497 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:13:17.718478 30607 consensus_peers.cc:485] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 -> Peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Couldn't send request to peer aad81047009a4ffcbc450b60030583b5. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:17.724013 30739 consensus_peers.cc:485] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 -> Peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Couldn't send request to peer aad81047009a4ffcbc450b60030583b5. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:17.735743 30739 consensus_peers.cc:485] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:17.755453 4734 raft_consensus.cc:482] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f1525cbc82a24d13b65f136695c903af)
I20240501 08:13:17.755640 4734 raft_consensus.cc:504] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:13:17.756290 4734 leader_election.cc:288] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
W20240501 08:13:17.757771 30473 leader_election.cc:334] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
W20240501 08:13:17.757987 30472 leader_election.cc:334] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
I20240501 08:13:17.758085 30472 leader_election.cc:302] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31811708c6774037b82df44024b229b6; no voters: 6331f77dae9d452184e8cd0cbf5f1b43, f1525cbc82a24d13b65f136695c903af
I20240501 08:13:17.758303 4774 raft_consensus.cc:2726] T ecab521101074a7b80b862642059daf2 P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:13:17.770699 3963 raft_consensus.cc:482] T 367847f920184dd6abcdcd7564f76376 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f1525cbc82a24d13b65f136695c903af)
I20240501 08:13:17.770884 3963 raft_consensus.cc:504] T 367847f920184dd6abcdcd7564f76376 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:13:17.771449 3963 leader_election.cc:288] T 367847f920184dd6abcdcd7564f76376 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
W20240501 08:13:17.771754 30739 consensus_peers.cc:485] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 -> Peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Couldn't send request to peer aad81047009a4ffcbc450b60030583b5. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:17.772945 30872 leader_election.cc:334] T 367847f920184dd6abcdcd7564f76376 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: connect: Connection refused (error 111)
W20240501 08:13:17.773885 30873 leader_election.cc:334] T 367847f920184dd6abcdcd7564f76376 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
I20240501 08:13:17.774003 30873 leader_election.cc:302] T 367847f920184dd6abcdcd7564f76376 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a29cd590999b4f32849ff5689cd53c11; no voters: aad81047009a4ffcbc450b60030583b5, f1525cbc82a24d13b65f136695c903af
I20240501 08:13:17.774199 3963 raft_consensus.cc:2726] T 367847f920184dd6abcdcd7564f76376 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:13:17.783784 4787 raft_consensus.cc:482] T b605254185d140d8ab90fcaa5229f168 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f1525cbc82a24d13b65f136695c903af)
I20240501 08:13:17.783975 4787 raft_consensus.cc:504] T b605254185d140d8ab90fcaa5229f168 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
I20240501 08:13:17.784894 4787 leader_election.cc:288] T b605254185d140d8ab90fcaa5229f168 P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:13:17.785444 4795 raft_consensus.cc:482] T 3aa09c722530431384e5e7ee9759230c P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:17.785604 4795 raft_consensus.cc:504] T 3aa09c722530431384e5e7ee9759230c P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:13:17.786406 4795 leader_election.cc:288] T 3aa09c722530431384e5e7ee9759230c P 91d11eae5ef6432ca86eb8ed446b6ea6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:13:17.790081 4778 raft_consensus.cc:482] T 928b70089c234425862c033341c82bad P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f1525cbc82a24d13b65f136695c903af)
I20240501 08:13:17.790238 4778 raft_consensus.cc:504] T 928b70089c234425862c033341c82bad P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } }
I20240501 08:13:17.790891 4778 leader_election.cc:288] T 928b70089c234425862c033341c82bad P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:13:17.792579 28151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fd595c79cdc43f9a8afadb6b548bf64" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:13:17.792771 28151 raft_consensus.cc:2454] T 4fd595c79cdc43f9a8afadb6b548bf64 P 91d11eae5ef6432ca86eb8ed446b6ea6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a022e1a501f4d44a0468054d6c3de47 in term 1.
I20240501 08:13:17.793378 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "928b70089c234425862c033341c82bad" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" is_pre_election: true
I20240501 08:13:17.793663 30671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b605254185d140d8ab90fcaa5229f168" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
W20240501 08:13:17.793920 30740 leader_election.cc:334] T 928b70089c234425862c033341c82bad P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
I20240501 08:13:17.794278 30605 leader_election.cc:302] T 4fd595c79cdc43f9a8afadb6b548bf64 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 91d11eae5ef6432ca86eb8ed446b6ea6, 9a022e1a501f4d44a0468054d6c3de47; no voters: f1525cbc82a24d13b65f136695c903af
I20240501 08:13:17.794833 4799 raft_consensus.cc:2781] T 4fd595c79cdc43f9a8afadb6b548bf64 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:13:17.794915 4799 raft_consensus.cc:482] T 4fd595c79cdc43f9a8afadb6b548bf64 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f1525cbc82a24d13b65f136695c903af)
I20240501 08:13:17.794993 4799 raft_consensus.cc:3037] T 4fd595c79cdc43f9a8afadb6b548bf64 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:17.794986 30737 leader_election.cc:302] T 928b70089c234425862c033341c82bad P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5; no voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af
I20240501 08:13:17.795403 4725 raft_consensus.cc:2726] T 928b70089c234425862c033341c82bad P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:13:17.797503 30800 raft_consensus.cc:1261] T e94895331d6b474484d987d42f13fd88 P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:13:17.798411 4777 consensus_queue.cc:1035] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:13:17.798208 4799 raft_consensus.cc:504] T 4fd595c79cdc43f9a8afadb6b548bf64 P 9a022e1a501f4d44a0468054d6c3de47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:13:17.803963 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 28070
W20240501 08:13:17.805514 30608 leader_election.cc:334] T 4fd595c79cdc43f9a8afadb6b548bf64 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
I20240501 08:13:17.806128 4799 leader_election.cc:288] T 4fd595c79cdc43f9a8afadb6b548bf64 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 election: Requested vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:13:17.807384 30535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aa09c722530431384e5e7ee9759230c" candidate_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:13:17.807564 30535 raft_consensus.cc:2454] T 3aa09c722530431384e5e7ee9759230c P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91d11eae5ef6432ca86eb8ed446b6ea6 in term 1.
I20240501 08:13:17.807778 3963 raft_consensus.cc:482] T 2518578063474b668508d644a2098331 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:17.807899 3963 raft_consensus.cc:504] T 2518578063474b668508d644a2098331 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:13:17.808413 3963 leader_election.cc:288] T 2518578063474b668508d644a2098331 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:13:17.808527 3963 raft_consensus.cc:482] T 019d8108d6d84d608fd8d5237f3f20ff P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:17.808599 3963 raft_consensus.cc:504] T 019d8108d6d84d608fd8d5237f3f20ff P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:13:17.809012 3963 leader_election.cc:288] T 019d8108d6d84d608fd8d5237f3f20ff P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:13:17.809486 4779 raft_consensus.cc:482] T a05223cb2f20478b8d29ffa065b4c67b P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
W20240501 08:13:17.809579 30472 consensus_peers.cc:485] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:17.809572 4779 raft_consensus.cc:504] T a05223cb2f20478b8d29ffa065b4c67b P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:13:17.809898 4779 leader_election.cc:288] T a05223cb2f20478b8d29ffa065b4c67b P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
W20240501 08:13:17.811807 30872 leader_election.cc:334] T 019d8108d6d84d608fd8d5237f3f20ff P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
W20240501 08:13:17.811930 30872 leader_election.cc:334] T a05223cb2f20478b8d29ffa065b4c67b P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
W20240501 08:13:17.812018 30872 leader_election.cc:334] T 2518578063474b668508d644a2098331 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
W20240501 08:13:17.812811 30872 leader_election.cc:334] T a05223cb2f20478b8d29ffa065b4c67b P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: connect: Connection refused (error 111)
I20240501 08:13:17.812928 30872 leader_election.cc:302] T a05223cb2f20478b8d29ffa065b4c67b P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a29cd590999b4f32849ff5689cd53c11; no voters: 6331f77dae9d452184e8cd0cbf5f1b43, aad81047009a4ffcbc450b60030583b5
I20240501 08:13:17.813102 30537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2518578063474b668508d644a2098331" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:13:17.813302 4779 raft_consensus.cc:2726] T a05223cb2f20478b8d29ffa065b4c67b P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:13:17.813318 30537 raft_consensus.cc:2454] T 2518578063474b668508d644a2098331 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:13:17.813921 30535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019d8108d6d84d608fd8d5237f3f20ff" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:13:17.814167 30535 raft_consensus.cc:2454] T 019d8108d6d84d608fd8d5237f3f20ff P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:13:17.815913 30873 leader_election.cc:302] T 2518578063474b668508d644a2098331 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters: 6331f77dae9d452184e8cd0cbf5f1b43
I20240501 08:13:17.816282 30873 leader_election.cc:302] T 019d8108d6d84d608fd8d5237f3f20ff P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters: 6331f77dae9d452184e8cd0cbf5f1b43
I20240501 08:13:17.816598 4779 raft_consensus.cc:2781] T 2518578063474b668508d644a2098331 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:13:17.816663 4779 raft_consensus.cc:482] T 2518578063474b668508d644a2098331 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:17.816743 4779 raft_consensus.cc:3037] T 2518578063474b668508d644a2098331 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:17.818877 4779 raft_consensus.cc:504] T 2518578063474b668508d644a2098331 P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:13:17.819509 4779 leader_election.cc:288] T 2518578063474b668508d644a2098331 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 election: Requested vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:13:17.819653 4779 raft_consensus.cc:2781] T 019d8108d6d84d608fd8d5237f3f20ff P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:13:17.819710 4779 raft_consensus.cc:482] T 019d8108d6d84d608fd8d5237f3f20ff P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:17.819768 4779 raft_consensus.cc:3037] T 019d8108d6d84d608fd8d5237f3f20ff P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Advancing to term 2
W20240501 08:13:17.821024 30872 leader_election.cc:334] T 2518578063474b668508d644a2098331 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
I20240501 08:13:17.822297 30535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2518578063474b668508d644a2098331" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:13:17.822494 30535 raft_consensus.cc:3037] T 2518578063474b668508d644a2098331 P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:17.824151 30535 raft_consensus.cc:2454] T 2518578063474b668508d644a2098331 P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 2.
I20240501 08:13:17.825336 30873 leader_election.cc:302] T 2518578063474b668508d644a2098331 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters: 6331f77dae9d452184e8cd0cbf5f1b43
I20240501 08:13:17.825713 3963 raft_consensus.cc:2781] T 2518578063474b668508d644a2098331 P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:17.825820 3963 raft_consensus.cc:686] T 2518578063474b668508d644a2098331 P a29cd590999b4f32849ff5689cd53c11 [term 2 LEADER]: Becoming Leader. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:13:17.826088 3963 consensus_queue.cc:237] T 2518578063474b668508d644a2098331 P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } }
I20240501 08:13:17.828152 4779 raft_consensus.cc:504] T 019d8108d6d84d608fd8d5237f3f20ff P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:13:17.829418 30535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019d8108d6d84d608fd8d5237f3f20ff" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:13:17.829691 30535 raft_consensus.cc:3037] T 019d8108d6d84d608fd8d5237f3f20ff P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:17.830605 30802 raft_consensus.cc:1261] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240501 08:13:17.830602 30872 leader_election.cc:334] T 019d8108d6d84d608fd8d5237f3f20ff P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
I20240501 08:13:17.831187 4776 consensus_queue.cc:1035] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:17.833150 30535 raft_consensus.cc:2454] T 019d8108d6d84d608fd8d5237f3f20ff P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 2.
W20240501 08:13:17.834465 30872 consensus_peers.cc:485] T 0481654b2e3c45e98fe51845319c25ee P a29cd590999b4f32849ff5689cd53c11 -> Peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Couldn't send request to peer aad81047009a4ffcbc450b60030583b5. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:17.834733 30873 leader_election.cc:302] T 019d8108d6d84d608fd8d5237f3f20ff P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters: 6331f77dae9d452184e8cd0cbf5f1b43
I20240501 08:13:17.834980 4727 raft_consensus.cc:2781] T 019d8108d6d84d608fd8d5237f3f20ff P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Leader election won for term 2
W20240501 08:13:17.835652 30472 consensus_peers.cc:485] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:17.838111 30607 consensus_peers.cc:485] T e94895331d6b474484d987d42f13fd88 P 9a022e1a501f4d44a0468054d6c3de47 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:17.838938 4779 leader_election.cc:288] T 019d8108d6d84d608fd8d5237f3f20ff P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 election: Requested vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:13:17.839820 4727 raft_consensus.cc:686] T 019d8108d6d84d608fd8d5237f3f20ff P a29cd590999b4f32849ff5689cd53c11 [term 2 LEADER]: Becoming Leader. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:13:17.840147 4727 consensus_queue.cc:237] T 019d8108d6d84d608fd8d5237f3f20ff P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:13:17.842918 27612 catalog_manager.cc:5484] T 2518578063474b668508d644a2098331 P a29cd590999b4f32849ff5689cd53c11 reported cstate change: term changed from 1 to 2, leader changed from 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) to a29cd590999b4f32849ff5689cd53c11 (127.21.84.136). New cstate: current_term: 2 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } }
W20240501 08:13:17.849608 30607 consensus_peers.cc:485] T 66b8b9c025b941a1a1b6e681fa0bbe90 P 9a022e1a501f4d44a0468054d6c3de47 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:17.850966 4727 raft_consensus.cc:482] T 948f6afb45594870bfe44b45d303f88f P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f1525cbc82a24d13b65f136695c903af)
I20240501 08:13:17.851585 4727 raft_consensus.cc:504] T 948f6afb45594870bfe44b45d303f88f P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:13:17.852329 4727 leader_election.cc:288] T 948f6afb45594870bfe44b45d303f88f P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:13:17.852770 30535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "948f6afb45594870bfe44b45d303f88f" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:13:17.852953 30535 raft_consensus.cc:2454] T 948f6afb45594870bfe44b45d303f88f P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:13:17.853366 30873 leader_election.cc:302] T 948f6afb45594870bfe44b45d303f88f P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:13:17.853648 4727 raft_consensus.cc:2781] T 948f6afb45594870bfe44b45d303f88f P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:13:17.853714 4727 raft_consensus.cc:482] T 948f6afb45594870bfe44b45d303f88f P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f1525cbc82a24d13b65f136695c903af)
I20240501 08:13:17.853787 4727 raft_consensus.cc:3037] T 948f6afb45594870bfe44b45d303f88f P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:17.853716 27619 catalog_manager.cc:5484] T 019d8108d6d84d608fd8d5237f3f20ff P a29cd590999b4f32849ff5689cd53c11 reported cstate change: term changed from 1 to 2, leader changed from 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129) to a29cd590999b4f32849ff5689cd53c11 (127.21.84.136). New cstate: current_term: 2 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } }
I20240501 08:13:17.855731 4727 raft_consensus.cc:504] T 948f6afb45594870bfe44b45d303f88f P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:13:17.856392 4727 leader_election.cc:288] T 948f6afb45594870bfe44b45d303f88f P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 election: Requested vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:13:17.856937 30535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "948f6afb45594870bfe44b45d303f88f" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6"
I20240501 08:13:17.857120 30535 raft_consensus.cc:3037] T 948f6afb45594870bfe44b45d303f88f P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Advancing to term 2
W20240501 08:13:17.859401 30873 leader_election.cc:334] T 948f6afb45594870bfe44b45d303f88f P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
I20240501 08:13:17.859705 30535 raft_consensus.cc:2454] T 948f6afb45594870bfe44b45d303f88f P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 2.
I20240501 08:13:17.860234 30873 leader_election.cc:302] T 948f6afb45594870bfe44b45d303f88f P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters:
W20240501 08:13:17.860491 30873 leader_election.cc:334] T 948f6afb45594870bfe44b45d303f88f P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
I20240501 08:13:17.860780 4727 raft_consensus.cc:2781] T 948f6afb45594870bfe44b45d303f88f P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:17.860900 4727 raft_consensus.cc:686] T 948f6afb45594870bfe44b45d303f88f P a29cd590999b4f32849ff5689cd53c11 [term 2 LEADER]: Becoming Leader. State: Replica: a29cd590999b4f32849ff5689cd53c11, State: Running, Role: LEADER
I20240501 08:13:17.861124 4727 consensus_queue.cc:237] T 948f6afb45594870bfe44b45d303f88f P a29cd590999b4f32849ff5689cd53c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:13:17.866838 27613 catalog_manager.cc:5484] T 948f6afb45594870bfe44b45d303f88f P a29cd590999b4f32849ff5689cd53c11 reported cstate change: term changed from 1 to 2, leader changed from f1525cbc82a24d13b65f136695c903af (127.21.84.130) to a29cd590999b4f32849ff5689cd53c11 (127.21.84.136). New cstate: current_term: 2 leader_uuid: "a29cd590999b4f32849ff5689cd53c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:17.885049 4774 raft_consensus.cc:482] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f1525cbc82a24d13b65f136695c903af)
I20240501 08:13:17.885239 4774 raft_consensus.cc:504] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:13:17.885808 4774 leader_election.cc:288] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:13:17.886322 30934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05eac5f1b08945f8b7b8c78d0f4d3d3a" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:13:17.886525 30934 raft_consensus.cc:2454] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 1.
I20240501 08:13:17.887174 30472 leader_election.cc:302] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:13:17.887404 4774 raft_consensus.cc:2781] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240501 08:13:17.887409 30473 leader_election.cc:334] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
I20240501 08:13:17.887497 4774 raft_consensus.cc:482] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f1525cbc82a24d13b65f136695c903af)
I20240501 08:13:17.887564 4774 raft_consensus.cc:3037] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Advancing to term 2
W20240501 08:13:17.888600 30871 connection.cc:535] server connection from 127.21.84.132:58509 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:17.888819 30605 connection.cc:535] client connection to 127.21.84.132:41323 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:17.889125 30470 connection.cc:535] client connection to 127.21.84.132:41323 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:17.889376 30737 connection.cc:535] client connection to 127.21.84.132:41323 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:13:17.889954 4774 raft_consensus.cc:504] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
W20240501 08:13:17.890442 27606 connection.cc:535] server connection from 127.21.84.132:34149 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:13:17.890566 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 30454
W20240501 08:13:17.891112 30870 connection.cc:535] client connection to 127.21.84.132:41323 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:17.891774 30605 leader_election.cc:334] T 4fd595c79cdc43f9a8afadb6b548bf64 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Network error: Client connection negotiation failed: client connection to 127.21.84.132:41323: connect: Connection refused (error 111)
W20240501 08:13:17.891829 30737 consensus_peers.cc:485] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 -> Peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Couldn't send request to peer 91d11eae5ef6432ca86eb8ed446b6ea6. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.132:41323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:17.891937 30737 consensus_peers.cc:485] T 00205a461b704a19bbd39bc71361e529 P 64998323a1f4482d8704bf20594d13a5 -> Peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Couldn't send request to peer 91d11eae5ef6432ca86eb8ed446b6ea6. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.132:41323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:17.891902 30605 leader_election.cc:302] T 4fd595c79cdc43f9a8afadb6b548bf64 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a022e1a501f4d44a0468054d6c3de47; no voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af
W20240501 08:13:17.891996 30737 consensus_peers.cc:485] T 9df818d6cb4946989f8a4d2e2ce9b8bb P 64998323a1f4482d8704bf20594d13a5 -> Peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Couldn't send request to peer 91d11eae5ef6432ca86eb8ed446b6ea6. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.132:41323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:17.892048 30737 consensus_peers.cc:485] T c51625843b8d44a8879fa2fdbecbee5f P 64998323a1f4482d8704bf20594d13a5 -> Peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Couldn't send request to peer 91d11eae5ef6432ca86eb8ed446b6ea6. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.132:41323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:17.892318 4777 raft_consensus.cc:2726] T 4fd595c79cdc43f9a8afadb6b548bf64 P 9a022e1a501f4d44a0468054d6c3de47 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 08:13:17.892573 4774 leader_election.cc:288] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 election: Requested vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
W20240501 08:13:17.890568 30470 consensus_peers.cc:485] T 893c1555ca0849da8f48a4f253b42153 P 31811708c6774037b82df44024b229b6 -> Peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Couldn't send request to peer 91d11eae5ef6432ca86eb8ed446b6ea6. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.132:41323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:17.890914 30472 connection.cc:535] server connection from 127.21.84.132:60719 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:17.892706 30470 leader_election.cc:334] T 3aa09c722530431384e5e7ee9759230c P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Network error: Client connection negotiation failed: client connection to 127.21.84.132:41323: connect: Connection refused (error 111)
I20240501 08:13:17.892787 30470 leader_election.cc:302] T 3aa09c722530431384e5e7ee9759230c P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31811708c6774037b82df44024b229b6; no voters: 6331f77dae9d452184e8cd0cbf5f1b43, 91d11eae5ef6432ca86eb8ed446b6ea6
W20240501 08:13:17.893043 30470 consensus_peers.cc:485] T 80bb0e53796f4b319fa9f877451076f3 P 31811708c6774037b82df44024b229b6 -> Peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Couldn't send request to peer 91d11eae5ef6432ca86eb8ed446b6ea6. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.132:41323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:17.893123 30470 consensus_peers.cc:485] T f63215605ade41a5bc8d97a35ce33a31 P 31811708c6774037b82df44024b229b6 -> Peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Couldn't send request to peer 91d11eae5ef6432ca86eb8ed446b6ea6. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.132:41323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:17.893209 30470 consensus_peers.cc:485] T d020e1225d5d447f993f920569a05f3d P 31811708c6774037b82df44024b229b6 -> Peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Couldn't send request to peer 91d11eae5ef6432ca86eb8ed446b6ea6. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.132:41323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:17.893379 4774 raft_consensus.cc:2726] T 3aa09c722530431384e5e7ee9759230c P 31811708c6774037b82df44024b229b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:13:17.893486 30934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05eac5f1b08945f8b7b8c78d0f4d3d3a" candidate_uuid: "31811708c6774037b82df44024b229b6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:13:17.893662 30934 raft_consensus.cc:3037] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Advancing to term 2
W20240501 08:13:17.894482 30870 leader_election.cc:334] T a2981c0863bc41729605f8b076b6a6cb P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Network error: Client connection negotiation failed: client connection to 127.21.84.132:41323: connect: Connection refused (error 111)
I20240501 08:13:17.894587 30870 leader_election.cc:302] T a2981c0863bc41729605f8b076b6a6cb P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a29cd590999b4f32849ff5689cd53c11; no voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af
W20240501 08:13:17.894853 30870 consensus_peers.cc:485] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 -> Peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Couldn't send request to peer 91d11eae5ef6432ca86eb8ed446b6ea6. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.132:41323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:17.894937 30870 leader_election.cc:334] T f3b6691cbe8740d2978c20209c78f3ff P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Network error: Client connection negotiation failed: client connection to 127.21.84.132:41323: connect: Connection refused (error 111)
I20240501 08:13:17.895021 30870 leader_election.cc:302] T f3b6691cbe8740d2978c20209c78f3ff P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a29cd590999b4f32849ff5689cd53c11; no voters: 6331f77dae9d452184e8cd0cbf5f1b43, 91d11eae5ef6432ca86eb8ed446b6ea6
W20240501 08:13:17.895251 30870 consensus_peers.cc:485] T d3673fe03fb64c47b30dc2b79042f44a P a29cd590999b4f32849ff5689cd53c11 -> Peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Couldn't send request to peer 91d11eae5ef6432ca86eb8ed446b6ea6. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.132:41323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:17.895344 30870 consensus_peers.cc:485] T 8ab0410a98974002ab68dd0f6a456d59 P a29cd590999b4f32849ff5689cd53c11 -> Peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Couldn't send request to peer 91d11eae5ef6432ca86eb8ed446b6ea6. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.132:41323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:17.895413 30870 consensus_peers.cc:485] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 -> Peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Couldn't send request to peer 91d11eae5ef6432ca86eb8ed446b6ea6. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.132:41323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:17.895471 30870 consensus_peers.cc:485] T 6c08d494131240c1ae8180f652208325 P a29cd590999b4f32849ff5689cd53c11 -> Peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Couldn't send request to peer 91d11eae5ef6432ca86eb8ed446b6ea6. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.132:41323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:17.895524 30870 consensus_peers.cc:485] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 -> Peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Couldn't send request to peer 91d11eae5ef6432ca86eb8ed446b6ea6. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.132:41323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:17.895579 30870 consensus_peers.cc:485] T 98525304f2334bf0a3f52c06df58b3ba P a29cd590999b4f32849ff5689cd53c11 -> Peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Couldn't send request to peer 91d11eae5ef6432ca86eb8ed446b6ea6. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.132:41323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:17.895777 4823 raft_consensus.cc:2726] T a2981c0863bc41729605f8b076b6a6cb P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:13:17.895895 30473 leader_election.cc:334] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
I20240501 08:13:17.895923 4823 raft_consensus.cc:2726] T f3b6691cbe8740d2978c20209c78f3ff P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:13:17.896174 30934 raft_consensus.cc:2454] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31811708c6774037b82df44024b229b6 in term 2.
I20240501 08:13:17.896610 30472 leader_election.cc:302] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P 31811708c6774037b82df44024b229b6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31811708c6774037b82df44024b229b6, a29cd590999b4f32849ff5689cd53c11; no voters: f1525cbc82a24d13b65f136695c903af
I20240501 08:13:17.896857 4774 raft_consensus.cc:2781] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:17.896951 4774 raft_consensus.cc:686] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P 31811708c6774037b82df44024b229b6 [term 2 LEADER]: Becoming Leader. State: Replica: 31811708c6774037b82df44024b229b6, State: Running, Role: LEADER
I20240501 08:13:17.897154 4774 consensus_queue.cc:237] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P 31811708c6774037b82df44024b229b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
W20240501 08:13:17.899000 30870 consensus_peers.cc:485] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 -> Peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Couldn't send request to peer 91d11eae5ef6432ca86eb8ed446b6ea6. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.132:41323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:17.909590 27613 catalog_manager.cc:5484] T 05eac5f1b08945f8b7b8c78d0f4d3d3a P 31811708c6774037b82df44024b229b6 reported cstate change: term changed from 1 to 2, leader changed from f1525cbc82a24d13b65f136695c903af (127.21.84.130) to 31811708c6774037b82df44024b229b6 (127.21.84.133). New cstate: current_term: 2 leader_uuid: "31811708c6774037b82df44024b229b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:17.911316 30578 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 08:13:17.928968 30605 consensus_peers.cc:485] T cb6e886e8cdd451d9a192a771482728e P 9a022e1a501f4d44a0468054d6c3de47 -> Peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Couldn't send request to peer 91d11eae5ef6432ca86eb8ed446b6ea6. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.132:41323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:17.931208 4740 raft_consensus.cc:482] T c4851258508d46c88493503a6257f365 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f1525cbc82a24d13b65f136695c903af)
I20240501 08:13:17.931380 4740 raft_consensus.cc:504] T c4851258508d46c88493503a6257f365 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } attrs { promote: false } }
I20240501 08:13:17.932032 4740 leader_election.cc:288] T c4851258508d46c88493503a6257f365 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
W20240501 08:13:17.933599 30740 leader_election.cc:334] T c4851258508d46c88493503a6257f365 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
W20240501 08:13:17.935022 30737 leader_election.cc:334] T c4851258508d46c88493503a6257f365 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Network error: Client connection negotiation failed: client connection to 127.21.84.132:41323: connect: Connection refused (error 111)
I20240501 08:13:17.935137 30737 leader_election.cc:302] T c4851258508d46c88493503a6257f365 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5; no voters: 91d11eae5ef6432ca86eb8ed446b6ea6, f1525cbc82a24d13b65f136695c903af
I20240501 08:13:17.935487 4740 raft_consensus.cc:2726] T c4851258508d46c88493503a6257f365 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:13:17.947544 30737 consensus_peers.cc:485] T 728616fd583a4afea00e8fe52880e55b P 64998323a1f4482d8704bf20594d13a5 -> Peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Couldn't send request to peer 91d11eae5ef6432ca86eb8ed446b6ea6. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.132:41323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:17.965378 30739 consensus_peers.cc:485] T 9b5b0b1e46d14d78806859541e0ab3b6 P 64998323a1f4482d8704bf20594d13a5 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:17.978293 30607 consensus_peers.cc:485] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:18.054497 30873 consensus_peers.cc:485] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:18.074057 30737 consensus_peers.cc:485] T a1fde6b765374df9bc4674a5d678c920 P 64998323a1f4482d8704bf20594d13a5 -> Peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Couldn't send request to peer 91d11eae5ef6432ca86eb8ed446b6ea6. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.132:41323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:18.107774 4777 raft_consensus.cc:482] T 6c04880ea87d4894bd3a63d94f4a69f4 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aad81047009a4ffcbc450b60030583b5)
I20240501 08:13:18.107976 4777 raft_consensus.cc:504] T 6c04880ea87d4894bd3a63d94f4a69f4 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:13:18.108668 4777 leader_election.cc:288] T 6c04880ea87d4894bd3a63d94f4a69f4 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
W20240501 08:13:18.109966 30607 leader_election.cc:334] T 6c04880ea87d4894bd3a63d94f4a69f4 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: connect: Connection refused (error 111)
W20240501 08:13:18.206610 30737 consensus_peers.cc:485] T b6ad187384514267b57cc994c9612709 P 64998323a1f4482d8704bf20594d13a5 -> Peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Couldn't send request to peer 91d11eae5ef6432ca86eb8ed446b6ea6. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.132:41323: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:18.235515 30537 raft_consensus.cc:1261] T 802538381de843ab8d326014a0b4ec15 P 31811708c6774037b82df44024b229b6 [term 2 FOLLOWER]: Refusing update from remote peer 9a022e1a501f4d44a0468054d6c3de47: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 08:13:18.236809 30802 raft_consensus.cc:1261] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 64998323a1f4482d8704bf20594d13a5 [term 3 FOLLOWER]: Refusing update from remote peer 31811708c6774037b82df44024b229b6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240501 08:13:18.237221 30534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c04880ea87d4894bd3a63d94f4a69f4" candidate_uuid: "9a022e1a501f4d44a0468054d6c3de47" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "31811708c6774037b82df44024b229b6" is_pre_election: true
I20240501 08:13:18.237985 4832 consensus_queue.cc:1035] T 73f40a7ef6954b34a48d8c5aeeaebc0a P 31811708c6774037b82df44024b229b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:13:18.240222 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 30454
I20240501 08:13:18.243993 4799 consensus_queue.cc:1035] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 08:13:18.244467 30608 leader_election.cc:302] T 6c04880ea87d4894bd3a63d94f4a69f4 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a022e1a501f4d44a0468054d6c3de47; no voters: 31811708c6774037b82df44024b229b6, aad81047009a4ffcbc450b60030583b5
I20240501 08:13:18.244689 4799 raft_consensus.cc:2726] T 6c04880ea87d4894bd3a63d94f4a69f4 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:13:18.299856 30740 connection.cc:535] server connection from 127.21.84.133:58883 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:18.300122 30740 connection.cc:535] client connection to 127.21.84.133:38293 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:18.300179 30873 connection.cc:535] client connection to 127.21.84.133:38293 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:18.300384 30871 connection.cc:535] server connection from 127.21.84.133:34677 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:18.300484 30608 connection.cc:535] client connection to 127.21.84.133:38293 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:13:18.300666 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 30588
W20240501 08:13:18.301532 30740 consensus_peers.cc:485] T 9b5b0b1e46d14d78806859541e0ab3b6 P 64998323a1f4482d8704bf20594d13a5 -> Peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): Couldn't send request to peer 31811708c6774037b82df44024b229b6. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.133:38293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:18.301651 30740 consensus_peers.cc:485] T bc84d1d413994a76bd6cf42335438755 P 64998323a1f4482d8704bf20594d13a5 -> Peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): Couldn't send request to peer 31811708c6774037b82df44024b229b6. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.133:38293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:18.301781 30608 consensus_peers.cc:485] T 802538381de843ab8d326014a0b4ec15 P 9a022e1a501f4d44a0468054d6c3de47 -> Peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): Couldn't send request to peer 31811708c6774037b82df44024b229b6. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.133:38293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:18.302099 30873 consensus_peers.cc:485] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 -> Peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): Couldn't send request to peer 31811708c6774037b82df44024b229b6. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.133:38293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:18.302224 30873 consensus_peers.cc:485] T 7f7211c8c0e041bf89930aa50a4438ea P a29cd590999b4f32849ff5689cd53c11 -> Peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): Couldn't send request to peer 31811708c6774037b82df44024b229b6. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.133:38293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:18.302325 30873 consensus_peers.cc:485] T 019d8108d6d84d608fd8d5237f3f20ff P a29cd590999b4f32849ff5689cd53c11 -> Peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): Couldn't send request to peer 31811708c6774037b82df44024b229b6. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.133:38293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:18.302413 30873 consensus_peers.cc:485] T a1965bd993d748f6a33cf93e8a3d70bf P a29cd590999b4f32849ff5689cd53c11 -> Peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): Couldn't send request to peer 31811708c6774037b82df44024b229b6. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.133:38293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:18.307130 4725 raft_consensus.cc:482] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f1525cbc82a24d13b65f136695c903af)
I20240501 08:13:18.307310 4725 raft_consensus.cc:504] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:13:18.307492 30713 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:13:18.308108 4725 leader_election.cc:288] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:13:18.308154 4778 raft_consensus.cc:482] T b82bf9ba2a7141cc81c6767cdc774c3a P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f1525cbc82a24d13b65f136695c903af)
I20240501 08:13:18.308287 4778 raft_consensus.cc:504] T b82bf9ba2a7141cc81c6767cdc774c3a P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:13:18.308699 4740 raft_consensus.cc:482] T 9820951bffc749139229e2b79c8ab597 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f1525cbc82a24d13b65f136695c903af)
I20240501 08:13:18.308815 4740 raft_consensus.cc:504] T 9820951bffc749139229e2b79c8ab597 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:13:18.308835 30934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c8e3a6f66a42b08507668f7c3a8af1" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:13:18.308818 4778 leader_election.cc:288] T b82bf9ba2a7141cc81c6767cdc774c3a P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:13:18.309062 4778 raft_consensus.cc:482] T 0be6dad1a409426c9a3dc0866505b3ea P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f1525cbc82a24d13b65f136695c903af)
I20240501 08:13:18.309085 30934 raft_consensus.cc:2454] T 98c8e3a6f66a42b08507668f7c3a8af1 P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 2.
I20240501 08:13:18.309151 4778 raft_consensus.cc:504] T 0be6dad1a409426c9a3dc0866505b3ea P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:13:18.309352 4740 leader_election.cc:288] T 9820951bffc749139229e2b79c8ab597 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
I20240501 08:13:18.309638 4778 leader_election.cc:288] T 0be6dad1a409426c9a3dc0866505b3ea P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:13:18.309685 30739 leader_election.cc:302] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:13:18.310222 30671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82bf9ba2a7141cc81c6767cdc774c3a" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:13:18.310302 4778 raft_consensus.cc:2781] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 08:13:18.310367 4778 raft_consensus.cc:482] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f1525cbc82a24d13b65f136695c903af)
I20240501 08:13:18.310425 4778 raft_consensus.cc:3037] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Advancing to term 3
W20240501 08:13:18.311280 30740 leader_election.cc:334] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
I20240501 08:13:18.312330 4778 raft_consensus.cc:504] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
W20240501 08:13:18.312656 30739 leader_election.cc:334] T 0be6dad1a409426c9a3dc0866505b3ea P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
I20240501 08:13:18.312798 4778 leader_election.cc:288] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 3 election: Requested vote from peers a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:13:18.313400 30934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c8e3a6f66a42b08507668f7c3a8af1" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11"
I20240501 08:13:18.313668 30934 raft_consensus.cc:3037] T 98c8e3a6f66a42b08507668f7c3a8af1 P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:18.317199 30934 raft_consensus.cc:2454] T 98c8e3a6f66a42b08507668f7c3a8af1 P a29cd590999b4f32849ff5689cd53c11 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 3.
I20240501 08:13:18.317616 30739 leader_election.cc:302] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:13:18.317854 4778 raft_consensus.cc:2781] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:13:18.317989 4778 raft_consensus.cc:686] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5 [term 3 LEADER]: Becoming Leader. State: Replica: 64998323a1f4482d8704bf20594d13a5, State: Running, Role: LEADER
I20240501 08:13:18.318248 4778 consensus_queue.cc:237] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
W20240501 08:13:18.317883 30740 leader_election.cc:334] T 9820951bffc749139229e2b79c8ab597 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
W20240501 08:13:18.318533 30740 leader_election.cc:334] T 0be6dad1a409426c9a3dc0866505b3ea P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
I20240501 08:13:18.318646 30740 leader_election.cc:302] T 0be6dad1a409426c9a3dc0866505b3ea P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5; no voters: 6331f77dae9d452184e8cd0cbf5f1b43, f1525cbc82a24d13b65f136695c903af
I20240501 08:13:18.318928 4725 raft_consensus.cc:2726] T 0be6dad1a409426c9a3dc0866505b3ea P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:13:18.319028 30740 leader_election.cc:334] T b82bf9ba2a7141cc81c6767cdc774c3a P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
I20240501 08:13:18.319137 30740 leader_election.cc:302] T b82bf9ba2a7141cc81c6767cdc774c3a P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5; no voters: 9a022e1a501f4d44a0468054d6c3de47, f1525cbc82a24d13b65f136695c903af
I20240501 08:13:18.319378 4725 raft_consensus.cc:2726] T b82bf9ba2a7141cc81c6767cdc774c3a P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:13:18.320480 30740 leader_election.cc:334] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
W20240501 08:13:18.325911 30873 consensus_peers.cc:485] T 82d6e5e399744b96a66373240ce1764d P a29cd590999b4f32849ff5689cd53c11 -> Peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): Couldn't send request to peer 31811708c6774037b82df44024b229b6. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.133:38293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:18.327726 27613 catalog_manager.cc:5484] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5 reported cstate change: term changed from 2 to 3, leader changed from f1525cbc82a24d13b65f136695c903af (127.21.84.130) to 64998323a1f4482d8704bf20594d13a5 (127.21.84.135). New cstate: current_term: 3 leader_uuid: "64998323a1f4482d8704bf20594d13a5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 08:13:18.333526 30872 consensus_peers.cc:485] T 2518578063474b668508d644a2098331 P a29cd590999b4f32849ff5689cd53c11 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:18.338917 30873 consensus_peers.cc:485] T 948f6afb45594870bfe44b45d303f88f P a29cd590999b4f32849ff5689cd53c11 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:18.344249 30873 consensus_peers.cc:485] T 6c08d494131240c1ae8180f652208325 P a29cd590999b4f32849ff5689cd53c11 -> Peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): Couldn't send request to peer 31811708c6774037b82df44024b229b6. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.133:38293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:18.348385 30872 consensus_peers.cc:485] T 019d8108d6d84d608fd8d5237f3f20ff P a29cd590999b4f32849ff5689cd53c11 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:18.365154 30740 consensus_peers.cc:485] T 32a54b70dcc447828ec293a9ff0afb80 P 64998323a1f4482d8704bf20594d13a5 -> Peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): Couldn't send request to peer 31811708c6774037b82df44024b229b6. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.133:38293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:18.393301 3963 raft_consensus.cc:482] T 5eccb9416d7d438893307f13dd4e73dc P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aad81047009a4ffcbc450b60030583b5)
I20240501 08:13:18.393499 3963 raft_consensus.cc:504] T 5eccb9416d7d438893307f13dd4e73dc P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:13:18.394624 3963 leader_election.cc:288] T 5eccb9416d7d438893307f13dd4e73dc P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
I20240501 08:13:18.395097 30802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eccb9416d7d438893307f13dd4e73dc" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
W20240501 08:13:18.395870 30872 leader_election.cc:334] T 5eccb9416d7d438893307f13dd4e73dc P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: connect: Connection refused (error 111)
I20240501 08:13:18.395977 30872 leader_election.cc:302] T 5eccb9416d7d438893307f13dd4e73dc P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a29cd590999b4f32849ff5689cd53c11; no voters: 64998323a1f4482d8704bf20594d13a5, aad81047009a4ffcbc450b60030583b5
I20240501 08:13:18.396178 3963 raft_consensus.cc:2726] T 5eccb9416d7d438893307f13dd4e73dc P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:13:18.416232 30873 consensus_peers.cc:485] T 2518578063474b668508d644a2098331 P a29cd590999b4f32849ff5689cd53c11 -> Peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): Couldn't send request to peer 31811708c6774037b82df44024b229b6. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.133:38293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:18.438269 30873 consensus_peers.cc:485] T 948f6afb45594870bfe44b45d303f88f P a29cd590999b4f32849ff5689cd53c11 -> Peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): Couldn't send request to peer 31811708c6774037b82df44024b229b6. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.133:38293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:18.443951 30873 consensus_peers.cc:485] T d3673fe03fb64c47b30dc2b79042f44a P a29cd590999b4f32849ff5689cd53c11 -> Peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): Couldn't send request to peer 31811708c6774037b82df44024b229b6. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.133:38293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:18.521211 30873 consensus_peers.cc:485] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 -> Peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): Couldn't send request to peer 31811708c6774037b82df44024b229b6. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.133:38293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:18.534358 30872 consensus_peers.cc:485] T 8bbd50f618d14aad8e45c2aa6fbabc1f P a29cd590999b4f32849ff5689cd53c11 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:13:18.620047 30872 consensus_peers.cc:485] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:13:18.661389 30740 consensus_peers.cc:485] T 647dba7e192140f9972724bd7e4a2569 P 64998323a1f4482d8704bf20594d13a5 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:13:18.663066 4777 raft_consensus.cc:482] T b82bf9ba2a7141cc81c6767cdc774c3a P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f1525cbc82a24d13b65f136695c903af)
I20240501 08:13:18.663542 30669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9820951bffc749139229e2b79c8ab597" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "9a022e1a501f4d44a0468054d6c3de47" is_pre_election: true
I20240501 08:13:18.663818 30669 raft_consensus.cc:2454] T 9820951bffc749139229e2b79c8ab597 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:13:18.664022 4776 raft_consensus.cc:482] T b605254185d140d8ab90fcaa5229f168 P 9a022e1a501f4d44a0468054d6c3de47 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f1525cbc82a24d13b65f136695c903af)
I20240501 08:13:18.664125 4776 raft_consensus.cc:504] T b605254185d140d8ab90fcaa5229f168 P 9a022e1a501f4d44a0468054d6c3de47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } attrs { promote: false } }
W20240501 08:13:18.664494 30605 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.84.132:41323, user_credentials={real_user=slave}} blocked reactor thread for 348376us
I20240501 08:13:18.664736 4776 leader_election.cc:288] T b605254185d140d8ab90fcaa5229f168 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:13:18.665253 4777 raft_consensus.cc:504] T b82bf9ba2a7141cc81c6767cdc774c3a P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } }
I20240501 08:13:18.665861 4777 leader_election.cc:288] T b82bf9ba2a7141cc81c6767cdc774c3a P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:13:18.665985 4799 raft_consensus.cc:482] T 52b69ecbe4094f32a1ed73b71b934451 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aad81047009a4ffcbc450b60030583b5)
I20240501 08:13:18.666095 4799 raft_consensus.cc:504] T 52b69ecbe4094f32a1ed73b71b934451 P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:13:18.666608 4799 leader_election.cc:288] T 52b69ecbe4094f32a1ed73b71b934451 P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
I20240501 08:13:18.666965 30737 leader_election.cc:302] T 9820951bffc749139229e2b79c8ab597 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, 9a022e1a501f4d44a0468054d6c3de47; no voters: f1525cbc82a24d13b65f136695c903af
I20240501 08:13:18.667549 4799 raft_consensus.cc:482] T e6fb30f10c754d3a8e73e124ec59768b P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aad81047009a4ffcbc450b60030583b5)
I20240501 08:13:18.667665 4799 raft_consensus.cc:504] T e6fb30f10c754d3a8e73e124ec59768b P 9a022e1a501f4d44a0468054d6c3de47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } attrs { promote: false } }
I20240501 08:13:18.668289 4799 leader_election.cc:288] T e6fb30f10c754d3a8e73e124ec59768b P 9a022e1a501f4d44a0468054d6c3de47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 31811708c6774037b82df44024b229b6 (127.21.84.133:38293)
I20240501 08:13:18.668696 4839 raft_consensus.cc:2781] T 9820951bffc749139229e2b79c8ab597 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:13:18.668787 4839 raft_consensus.cc:482] T 9820951bffc749139229e2b79c8ab597 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f1525cbc82a24d13b65f136695c903af)
I20240501 08:13:18.668857 4839 raft_consensus.cc:3037] T 9820951bffc749139229e2b79c8ab597 P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:18.670064 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 30588
I20240501 08:13:18.673199 4839 raft_consensus.cc:504] T 9820951bffc749139229e2b79c8ab597 P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:13:18.673976 4839 leader_election.cc:288] T 9820951bffc749139229e2b79c8ab597 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 election: Requested vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
W20240501 08:13:18.675086 30740 leader_election.cc:334] T 9820951bffc749139229e2b79c8ab597 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
W20240501 08:13:18.706707 30740 consensus_peers.cc:485] T fea1af7c0ecc4ddba679a981125d957a P 64998323a1f4482d8704bf20594d13a5 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:13:18.722096 30737 connection.cc:535] client connection to 127.21.84.134:39933 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:18.722096 30870 connection.cc:535] server connection from 127.21.84.134:36153 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:18.722316 30870 connection.cc:535] client connection to 127.21.84.134:39933 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:18.722307 27606 connection.cc:535] server connection from 127.21.84.134:59341 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:18.723471 30870 consensus_peers.cc:485] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 -> Peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Couldn't send request to peer 9a022e1a501f4d44a0468054d6c3de47. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.134:39933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:18.723572 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 30722
W20240501 08:13:18.723588 30870 consensus_peers.cc:485] T 0481654b2e3c45e98fe51845319c25ee P a29cd590999b4f32849ff5689cd53c11 -> Peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Couldn't send request to peer 9a022e1a501f4d44a0468054d6c3de47. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.134:39933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:18.723652 30870 consensus_peers.cc:485] T 22cdd207ff0c49e0b89d68de068d3603 P a29cd590999b4f32849ff5689cd53c11 -> Peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Couldn't send request to peer 9a022e1a501f4d44a0468054d6c3de47. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.134:39933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:18.723704 30870 consensus_peers.cc:485] T b6b0bcb4d07e4125bccf1711a094576e P a29cd590999b4f32849ff5689cd53c11 -> Peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Couldn't send request to peer 9a022e1a501f4d44a0468054d6c3de47. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.134:39933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:18.723673 30737 consensus_peers.cc:485] T fea1af7c0ecc4ddba679a981125d957a P 64998323a1f4482d8704bf20594d13a5 -> Peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Couldn't send request to peer 9a022e1a501f4d44a0468054d6c3de47. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.134:39933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:18.723754 30870 consensus_peers.cc:485] T 6c324b1136ce429290d7286399bb7121 P a29cd590999b4f32849ff5689cd53c11 -> Peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Couldn't send request to peer 9a022e1a501f4d44a0468054d6c3de47. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.134:39933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:18.723773 30737 consensus_peers.cc:485] T 325b3dbba80d491dab81840e8eb171e9 P 64998323a1f4482d8704bf20594d13a5 -> Peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Couldn't send request to peer 9a022e1a501f4d44a0468054d6c3de47. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.134:39933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:18.723814 30737 consensus_peers.cc:485] T 131eac1e19264e00b76500d2ba42697a P 64998323a1f4482d8704bf20594d13a5 -> Peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Couldn't send request to peer 9a022e1a501f4d44a0468054d6c3de47. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.134:39933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:18.723809 30870 consensus_peers.cc:485] T 8277d70d49f54197af60e0be86d34a9b P a29cd590999b4f32849ff5689cd53c11 -> Peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Couldn't send request to peer 9a022e1a501f4d44a0468054d6c3de47. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.134:39933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:18.723848 30737 consensus_peers.cc:485] T 40a4b09831f84433b6dc126b8445cf41 P 64998323a1f4482d8704bf20594d13a5 -> Peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Couldn't send request to peer 9a022e1a501f4d44a0468054d6c3de47. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.134:39933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:18.723858 30870 consensus_peers.cc:485] T 73058afba1214e21b655283c0612a0ec P a29cd590999b4f32849ff5689cd53c11 -> Peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Couldn't send request to peer 9a022e1a501f4d44a0468054d6c3de47. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.134:39933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:18.723889 30737 leader_election.cc:334] T 9820951bffc749139229e2b79c8ab597 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Network error: Client connection negotiation failed: client connection to 127.21.84.134:39933: connect: Connection refused (error 111)
W20240501 08:13:18.723907 30870 consensus_peers.cc:485] T 77513d2aea9246b4b37ab54e2a6b496a P a29cd590999b4f32849ff5689cd53c11 -> Peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Couldn't send request to peer 9a022e1a501f4d44a0468054d6c3de47. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.134:39933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:18.723955 30737 leader_election.cc:302] T 9820951bffc749139229e2b79c8ab597 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5; no voters: 9a022e1a501f4d44a0468054d6c3de47, f1525cbc82a24d13b65f136695c903af
I20240501 08:13:18.724216 4839 raft_consensus.cc:2726] T 9820951bffc749139229e2b79c8ab597 P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240501 08:13:18.724275 30737 consensus_peers.cc:485] T 2e1dd7a48a574ffcbea936af5b0fe192 P 64998323a1f4482d8704bf20594d13a5 -> Peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Couldn't send request to peer 9a022e1a501f4d44a0468054d6c3de47. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.134:39933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:18.724372 30737 consensus_peers.cc:485] T c51625843b8d44a8879fa2fdbecbee5f P 64998323a1f4482d8704bf20594d13a5 -> Peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Couldn't send request to peer 9a022e1a501f4d44a0468054d6c3de47. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.134:39933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:18.724436 30737 consensus_peers.cc:485] T fc9e0f35ebcd48b5a27c57b2485e599e P 64998323a1f4482d8704bf20594d13a5 -> Peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Couldn't send request to peer 9a022e1a501f4d44a0468054d6c3de47. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.134:39933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:13:18.724490 30737 consensus_peers.cc:485] T 00a73d23b12d48a2966f94789db4369a P 64998323a1f4482d8704bf20594d13a5 -> Peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Couldn't send request to peer 9a022e1a501f4d44a0468054d6c3de47. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.134:39933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:18.729125 30844 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 08:13:18.746693 30872 consensus_peers.cc:485] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:13:18.756021 30872 consensus_peers.cc:485] T 22cdd207ff0c49e0b89d68de068d3603 P a29cd590999b4f32849ff5689cd53c11 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:13:18.759075 30870 consensus_peers.cc:485] T 946add184f4a46f28fd631e8a3bc0ba7 P a29cd590999b4f32849ff5689cd53c11 -> Peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Couldn't send request to peer 9a022e1a501f4d44a0468054d6c3de47. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.134:39933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:18.774127 4837 raft_consensus.cc:482] T 28a7c90dfdf4459aae99c552cfba5b4b P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aad81047009a4ffcbc450b60030583b5)
I20240501 08:13:18.774084 4823 raft_consensus.cc:482] T 38c47b76fb304221b60d2b572485ad9f P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aad81047009a4ffcbc450b60030583b5)
I20240501 08:13:18.774278 4837 raft_consensus.cc:504] T 28a7c90dfdf4459aae99c552cfba5b4b P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:13:18.774279 4823 raft_consensus.cc:504] T 38c47b76fb304221b60d2b572485ad9f P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } }
I20240501 08:13:18.774044 3963 raft_consensus.cc:482] T b12559fc416a4f3cbffa018611226bcc P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aad81047009a4ffcbc450b60030583b5)
I20240501 08:13:18.774986 4823 leader_election.cc:288] T 38c47b76fb304221b60d2b572485ad9f P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:13:18.774986 4837 leader_election.cc:288] T 28a7c90dfdf4459aae99c552cfba5b4b P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:13:18.775084 3963 raft_consensus.cc:504] T b12559fc416a4f3cbffa018611226bcc P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "9a022e1a501f4d44a0468054d6c3de47" member_type: VOTER last_known_addr { host: "127.21.84.134" port: 39933 } attrs { promote: false } }
I20240501 08:13:18.775625 3963 leader_election.cc:288] T b12559fc416a4f3cbffa018611226bcc P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933)
W20240501 08:13:18.777035 30873 leader_election.cc:334] T 38c47b76fb304221b60d2b572485ad9f P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): Network error: Client connection negotiation failed: client connection to 127.21.84.133:38293: connect: Connection refused (error 111)
W20240501 08:13:18.777050 30872 leader_election.cc:334] T 28a7c90dfdf4459aae99c552cfba5b4b P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: connect: Connection refused (error 111)
W20240501 08:13:18.777200 30872 leader_election.cc:334] T b12559fc416a4f3cbffa018611226bcc P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: connect: Connection refused (error 111)
W20240501 08:13:18.777274 30870 leader_election.cc:334] T b12559fc416a4f3cbffa018611226bcc P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a022e1a501f4d44a0468054d6c3de47 (127.21.84.134:39933): Network error: Client connection negotiation failed: client connection to 127.21.84.134:39933: connect: Connection refused (error 111)
W20240501 08:13:18.777287 30872 leader_election.cc:334] T 38c47b76fb304221b60d2b572485ad9f P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: connect: Connection refused (error 111)
I20240501 08:13:18.777379 30870 leader_election.cc:302] T b12559fc416a4f3cbffa018611226bcc P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a29cd590999b4f32849ff5689cd53c11; no voters: 9a022e1a501f4d44a0468054d6c3de47, aad81047009a4ffcbc450b60030583b5
I20240501 08:13:18.777384 30872 leader_election.cc:302] T 38c47b76fb304221b60d2b572485ad9f P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a29cd590999b4f32849ff5689cd53c11; no voters: 31811708c6774037b82df44024b229b6, aad81047009a4ffcbc450b60030583b5
I20240501 08:13:18.777618 4837 raft_consensus.cc:2726] T 38c47b76fb304221b60d2b572485ad9f P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:13:18.777618 3963 raft_consensus.cc:2726] T b12559fc416a4f3cbffa018611226bcc P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:13:18.830971 3963 raft_consensus.cc:482] T 4b3e3d39f82549079293d98357884bf0 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 91d11eae5ef6432ca86eb8ed446b6ea6)
I20240501 08:13:18.831146 3963 raft_consensus.cc:504] T 4b3e3d39f82549079293d98357884bf0 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:13:18.831893 3963 leader_election.cc:288] T 4b3e3d39f82549079293d98357884bf0 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323), 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503)
W20240501 08:13:18.833513 30870 leader_election.cc:334] T 4b3e3d39f82549079293d98357884bf0 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Network error: Client connection negotiation failed: client connection to 127.21.84.132:41323: connect: Connection refused (error 111)
W20240501 08:13:18.871430 30872 consensus_peers.cc:485] T b5d7223054aa4308917fd402d65ded5d P a29cd590999b4f32849ff5689cd53c11 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:13:19.042304 4823 raft_consensus.cc:482] T 8c3d0b4ae0404bdfada8a86b23db9d30 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 31811708c6774037b82df44024b229b6)
I20240501 08:13:19.042526 4823 raft_consensus.cc:504] T 8c3d0b4ae0404bdfada8a86b23db9d30 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:13:19.043856 4778 raft_consensus.cc:482] T 30130709805e4fa6ae94a46034710863 P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader aad81047009a4ffcbc450b60030583b5)
I20240501 08:13:19.045506 4778 raft_consensus.cc:504] T 30130709805e4fa6ae94a46034710863 P 64998323a1f4482d8704bf20594d13a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } }
I20240501 08:13:19.045842 4725 raft_consensus.cc:482] T 5eccb9416d7d438893307f13dd4e73dc P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aad81047009a4ffcbc450b60030583b5)
I20240501 08:13:19.046039 4725 raft_consensus.cc:504] T 5eccb9416d7d438893307f13dd4e73dc P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:13:19.046376 4778 leader_election.cc:288] T 30130709805e4fa6ae94a46034710863 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:13:19.046751 4725 leader_election.cc:288] T 5eccb9416d7d438893307f13dd4e73dc P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:13:19.047159 4725 raft_consensus.cc:482] T 28a7c90dfdf4459aae99c552cfba5b4b P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aad81047009a4ffcbc450b60030583b5)
I20240501 08:13:19.047286 4725 raft_consensus.cc:504] T 28a7c90dfdf4459aae99c552cfba5b4b P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:13:19.047880 4725 leader_election.cc:288] T 28a7c90dfdf4459aae99c552cfba5b4b P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), a29cd590999b4f32849ff5689cd53c11 (127.21.84.136:46845)
I20240501 08:13:19.049212 30802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28a7c90dfdf4459aae99c552cfba5b4b" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:13:19.049472 30802 raft_consensus.cc:2454] T 28a7c90dfdf4459aae99c552cfba5b4b P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29cd590999b4f32849ff5689cd53c11 in term 1.
I20240501 08:13:19.050397 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b3e3d39f82549079293d98357884bf0" candidate_uuid: "a29cd590999b4f32849ff5689cd53c11" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "64998323a1f4482d8704bf20594d13a5" is_pre_election: true
I20240501 08:13:19.050515 30937 raft_consensus.cc:1261] T 98c8e3a6f66a42b08507668f7c3a8af1 P a29cd590999b4f32849ff5689cd53c11 [term 3 FOLLOWER]: Refusing update from remote peer 64998323a1f4482d8704bf20594d13a5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240501 08:13:19.050457 30934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eccb9416d7d438893307f13dd4e73dc" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:13:19.050791 30934 raft_consensus.cc:2454] T 5eccb9416d7d438893307f13dd4e73dc P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:13:19.050877 30936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28a7c90dfdf4459aae99c552cfba5b4b" candidate_uuid: "64998323a1f4482d8704bf20594d13a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a29cd590999b4f32849ff5689cd53c11" is_pre_election: true
I20240501 08:13:19.051040 30936 raft_consensus.cc:2454] T 28a7c90dfdf4459aae99c552cfba5b4b P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64998323a1f4482d8704bf20594d13a5 in term 1.
I20240501 08:13:19.053385 30871 leader_election.cc:302] T 28a7c90dfdf4459aae99c552cfba5b4b P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, a29cd590999b4f32849ff5689cd53c11; no voters: aad81047009a4ffcbc450b60030583b5
I20240501 08:13:19.053459 4823 leader_election.cc:288] T 8c3d0b4ae0404bdfada8a86b23db9d30 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:13:19.056802 30871 leader_election.cc:302] T 4b3e3d39f82549079293d98357884bf0 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a29cd590999b4f32849ff5689cd53c11; no voters: 64998323a1f4482d8704bf20594d13a5, 91d11eae5ef6432ca86eb8ed446b6ea6
I20240501 08:13:19.057581 3963 raft_consensus.cc:2781] T 28a7c90dfdf4459aae99c552cfba5b4b P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:13:19.057694 3963 raft_consensus.cc:482] T 28a7c90dfdf4459aae99c552cfba5b4b P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting leader election (detected failure of leader aad81047009a4ffcbc450b60030583b5)
I20240501 08:13:19.057793 3963 raft_consensus.cc:3037] T 28a7c90dfdf4459aae99c552cfba5b4b P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:19.061365 4837 raft_consensus.cc:2726] T 4b3e3d39f82549079293d98357884bf0 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:13:19.061823 3963 raft_consensus.cc:504] T 28a7c90dfdf4459aae99c552cfba5b4b P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:13:19.062359 4725 raft_consensus.cc:482] T b0906abad5c14912b94e9c017d2ecf8d P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6331f77dae9d452184e8cd0cbf5f1b43)
I20240501 08:13:19.062508 4725 raft_consensus.cc:504] T b0906abad5c14912b94e9c017d2ecf8d P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "64998323a1f4482d8704bf20594d13a5" member_type: VOTER last_known_addr { host: "127.21.84.135" port: 33503 } }
I20240501 08:13:19.063035 4725 leader_election.cc:288] T b0906abad5c14912b94e9c017d2ecf8d P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
I20240501 08:13:19.063055 3963 leader_election.cc:288] T 28a7c90dfdf4459aae99c552cfba5b4b P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 election: Requested vote from peers 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
W20240501 08:13:19.063227 30872 leader_election.cc:334] T 8c3d0b4ae0404bdfada8a86b23db9d30 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
W20240501 08:13:19.064482 30739 consensus_peers.cc:485] T f607d2e086cc4352a05a5ef2b02788cc P 64998323a1f4482d8704bf20594d13a5 -> Peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Couldn't send request to peer 6331f77dae9d452184e8cd0cbf5f1b43. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:13:19.064591 30739 leader_election.cc:334] T 30130709805e4fa6ae94a46034710863 P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
W20240501 08:13:19.064922 30872 leader_election.cc:334] T 28a7c90dfdf4459aae99c552cfba5b4b P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: connect: Connection refused (error 111)
I20240501 08:13:19.065304 30739 leader_election.cc:302] T 5eccb9416d7d438893307f13dd4e73dc P 64998323a1f4482d8704bf20594d13a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64998323a1f4482d8704bf20594d13a5, a29cd590999b4f32849ff5689cd53c11; no voters:
I20240501 08:13:19.065613 4740 consensus_queue.cc:1035] T 98c8e3a6f66a42b08507668f7c3a8af1 P 64998323a1f4482d8704bf20594d13a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:13:19.065790 4856 raft_consensus.cc:2781] T 5eccb9416d7d438893307f13dd4e73dc P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:13:19.065883 4856 raft_consensus.cc:482] T 5eccb9416d7d438893307f13dd4e73dc P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader aad81047009a4ffcbc450b60030583b5)
I20240501 08:13:19.066015 4856 raft_consensus.cc:3037] T 5eccb9416d7d438893307f13dd4e73dc P 64998323a1f4482d8704bf20594d13a5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:19.066386 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 30722
W20240501 08:13:19.069633 30873 leader_election.cc:334] T 8c3d0b4ae0404bdfada8a86b23db9d30 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): Network error: Client connection negotiation failed: client connection to 127.21.84.133:38293: connect: Connection refused (error 111)
I20240501 08:13:19.069793 30873 leader_election.cc:302] T 8c3d0b4ae0404bdfada8a86b23db9d30 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a29cd590999b4f32849ff5689cd53c11; no voters: 31811708c6774037b82df44024b229b6, 6331f77dae9d452184e8cd0cbf5f1b43
I20240501 08:13:19.070108 4857 raft_consensus.cc:2726] T 8c3d0b4ae0404bdfada8a86b23db9d30 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:13:19.101202 4857 raft_consensus.cc:482] T 83b422d7436a41e4aa6398963fb40f1e P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 31811708c6774037b82df44024b229b6)
I20240501 08:13:19.101418 4857 raft_consensus.cc:504] T 83b422d7436a41e4aa6398963fb40f1e P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } }
I20240501 08:13:19.102047 4857 leader_election.cc:288] T 83b422d7436a41e4aa6398963fb40f1e P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503)
W20240501 08:13:19.103523 30873 leader_election.cc:334] T 83b422d7436a41e4aa6398963fb40f1e P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): Network error: Client connection negotiation failed: client connection to 127.21.84.133:38293: connect: Connection refused (error 111)
W20240501 08:13:19.103706 30873 leader_election.cc:334] T 83b422d7436a41e4aa6398963fb40f1e P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
I20240501 08:13:19.103797 30873 leader_election.cc:302] T 83b422d7436a41e4aa6398963fb40f1e P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a29cd590999b4f32849ff5689cd53c11; no voters: 31811708c6774037b82df44024b229b6, f1525cbc82a24d13b65f136695c903af
I20240501 08:13:19.104038 4857 raft_consensus.cc:2726] T 83b422d7436a41e4aa6398963fb40f1e P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:13:19.118021 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 30853
W20240501 08:13:19.118001 27606 connection.cc:535] server connection from 127.21.84.135:53607 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:19.118260 30871 connection.cc:535] client connection to 127.21.84.135:33503 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:13:19.125224 30871 leader_election.cc:334] T 28a7c90dfdf4459aae99c552cfba5b4b P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 64998323a1f4482d8704bf20594d13a5 (127.21.84.135:33503): Network error: Client connection negotiation failed: client connection to 127.21.84.135:33503: connect: Connection refused (error 111)
I20240501 08:13:19.125386 30871 leader_election.cc:302] T 28a7c90dfdf4459aae99c552cfba5b4b P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a29cd590999b4f32849ff5689cd53c11; no voters: 64998323a1f4482d8704bf20594d13a5, aad81047009a4ffcbc450b60030583b5
I20240501 08:13:19.125831 4857 raft_consensus.cc:2726] T 28a7c90dfdf4459aae99c552cfba5b4b P a29cd590999b4f32849ff5689cd53c11 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 08:13:19.127854 30978 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:13:19.433413 4823 raft_consensus.cc:482] T 220add4033694ff381e023324a2a8b02 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 31811708c6774037b82df44024b229b6)
I20240501 08:13:19.434100 3963 raft_consensus.cc:482] T 1448b8f2d5844ad7aec9bcd3874c5c01 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 31811708c6774037b82df44024b229b6)
I20240501 08:13:19.434237 3963 raft_consensus.cc:504] T 1448b8f2d5844ad7aec9bcd3874c5c01 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } }
I20240501 08:13:19.434803 3963 leader_election.cc:288] T 1448b8f2d5844ad7aec9bcd3874c5c01 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
I20240501 08:13:19.433630 4823 raft_consensus.cc:504] T 220add4033694ff381e023324a2a8b02 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31811708c6774037b82df44024b229b6" member_type: VOTER last_known_addr { host: "127.21.84.133" port: 38293 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } } peers { permanent_uuid: "6331f77dae9d452184e8cd0cbf5f1b43" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 43471 } attrs { promote: false } }
I20240501 08:13:19.441577 4823 leader_election.cc:288] T 220add4033694ff381e023324a2a8b02 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31811708c6774037b82df44024b229b6 (127.21.84.133:38293), 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471)
I20240501 08:13:19.452059 4823 raft_consensus.cc:482] T 05b2485fa01a4706a85ebc150306637f P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 91d11eae5ef6432ca86eb8ed446b6ea6)
I20240501 08:13:19.452248 4823 raft_consensus.cc:504] T 05b2485fa01a4706a85ebc150306637f P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:13:19.452817 4823 leader_election.cc:288] T 05b2485fa01a4706a85ebc150306637f P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
W20240501 08:13:19.459710 30872 leader_election.cc:334] T 05b2485fa01a4706a85ebc150306637f P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: connect: Connection refused (error 111)
W20240501 08:13:19.459870 30872 leader_election.cc:334] T 1448b8f2d5844ad7aec9bcd3874c5c01 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305): Network error: Client connection negotiation failed: client connection to 127.21.84.131:36305: connect: Connection refused (error 111)
W20240501 08:13:19.462852 30872 leader_election.cc:334] T 220add4033694ff381e023324a2a8b02 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6331f77dae9d452184e8cd0cbf5f1b43 (127.21.84.129:43471): Network error: Client connection negotiation failed: client connection to 127.21.84.129:43471: connect: Connection refused (error 111)
I20240501 08:13:19.464864 4857 raft_consensus.cc:482] T 0e06cf862c954e8fb1ca087ec5184497 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f1525cbc82a24d13b65f136695c903af)
I20240501 08:13:19.465023 4857 raft_consensus.cc:504] T 0e06cf862c954e8fb1ca087ec5184497 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "aad81047009a4ffcbc450b60030583b5" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36305 } attrs { promote: false } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:13:19.465638 4857 leader_election.cc:288] T 0e06cf862c954e8fb1ca087ec5184497 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), aad81047009a4ffcbc450b60030583b5 (127.21.84.131:36305)
W20240501 08:13:19.466068 30873 leader_election.cc:334] T 220add4033694ff381e023324a2a8b02 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): Network error: Client connection negotiation failed: client connection to 127.21.84.133:38293: connect: Connection refused (error 111)
I20240501 08:13:19.466217 30873 leader_election.cc:302] T 220add4033694ff381e023324a2a8b02 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a29cd590999b4f32849ff5689cd53c11; no voters: 31811708c6774037b82df44024b229b6, 6331f77dae9d452184e8cd0cbf5f1b43
W20240501 08:13:19.466481 30873 leader_election.cc:334] T 1448b8f2d5844ad7aec9bcd3874c5c01 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 31811708c6774037b82df44024b229b6 (127.21.84.133:38293): Network error: Client connection negotiation failed: client connection to 127.21.84.133:38293: connect: Connection refused (error 111)
I20240501 08:13:19.466547 30873 leader_election.cc:302] T 1448b8f2d5844ad7aec9bcd3874c5c01 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a29cd590999b4f32849ff5689cd53c11; no voters: 31811708c6774037b82df44024b229b6, aad81047009a4ffcbc450b60030583b5
I20240501 08:13:19.466913 4866 raft_consensus.cc:2726] T 220add4033694ff381e023324a2a8b02 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:13:19.467068 4866 raft_consensus.cc:2726] T 1448b8f2d5844ad7aec9bcd3874c5c01 P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:13:19.467350 30873 consensus_peers.cc:485] T 4e58bc2dffad423f942ce50144030da4 P a29cd590999b4f32849ff5689cd53c11 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:13:19.467442 30873 consensus_peers.cc:485] T 87804550560d45269e6d4d6fe8938f55 P a29cd590999b4f32849ff5689cd53c11 -> Peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Couldn't send request to peer f1525cbc82a24d13b65f136695c903af. Status: Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:13:19.468684 30873 leader_election.cc:334] T 0e06cf862c954e8fb1ca087ec5184497 P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
I20240501 08:13:19.469136 4861 raft_consensus.cc:482] T a2981c0863bc41729605f8b076b6a6cb P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f1525cbc82a24d13b65f136695c903af)
I20240501 08:13:19.469292 4861 raft_consensus.cc:504] T a2981c0863bc41729605f8b076b6a6cb P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1525cbc82a24d13b65f136695c903af" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 36503 } } peers { permanent_uuid: "91d11eae5ef6432ca86eb8ed446b6ea6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 41323 } } peers { permanent_uuid: "a29cd590999b4f32849ff5689cd53c11" member_type: VOTER last_known_addr { host: "127.21.84.136" port: 46845 } attrs { promote: false } }
I20240501 08:13:19.469966 4861 leader_election.cc:288] T a2981c0863bc41729605f8b076b6a6cb P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503), 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323)
W20240501 08:13:19.471657 30870 leader_election.cc:334] T 05b2485fa01a4706a85ebc150306637f P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 91d11eae5ef6432ca86eb8ed446b6ea6 (127.21.84.132:41323): Network error: Client connection negotiation failed: client connection to 127.21.84.132:41323: connect: Connection refused (error 111)
I20240501 08:13:19.471753 30870 leader_election.cc:302] T 05b2485fa01a4706a85ebc150306637f P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a29cd590999b4f32849ff5689cd53c11; no voters: 91d11eae5ef6432ca86eb8ed446b6ea6, aad81047009a4ffcbc450b60030583b5
I20240501 08:13:19.471942 4861 raft_consensus.cc:2726] T 05b2485fa01a4706a85ebc150306637f P a29cd590999b4f32849ff5689cd53c11 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:13:19.472517 30873 leader_election.cc:334] T a2981c0863bc41729605f8b076b6a6cb P a29cd590999b4f32849ff5689cd53c11 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1525cbc82a24d13b65f136695c903af (127.21.84.130:36503): Network error: Client connection negotiation failed: client connection to 127.21.84.130:36503: connect: Connection refused (error 111)
I20240501 08:13:19.473270 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 30853
I20240501 08:13:19.517287 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 27589
I20240501 08:13:19.523089 27650 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:13:19.821208 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 27589
2024-05-01T08:13:19Z chronyd exiting
[ OK ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/1 (115753 ms)
[----------] 1 test from DDLDuringRebalancingTest (115753 ms total)
[----------] 1 test from RebalancingDuringElectionStormTest
[ RUN ] RebalancingDuringElectionStormTest.RoundRobin/1
I20240501 08:13:20.314545 21842 test_util.cc:268] Using random seed: 255751336
I20240501 08:13:20.316166 21842 ts_itest-base.cc:112] Starting cluster with:
I20240501 08:13:20.316215 21842 ts_itest-base.cc:113] --------------
I20240501 08:13:20.316232 21842 ts_itest-base.cc:114] 8 tablet servers
I20240501 08:13:20.316248 21842 ts_itest-base.cc:115] 3 replicas per TS
I20240501 08:13:20.316268 21842 ts_itest-base.cc:116] --------------
2024-05-01T08:13:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T08:13:20Z Disabled control of system clock
I20240501 08:13:20.360602 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.84.190:46613
--webserver_interface=127.21.84.190
--webserver_port=0
--builtin_ntp_servers=127.21.84.148:37541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.84.190:46613
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 08:13:20.730595 4879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:20.730933 4879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:13:20.731091 4879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:13:20.738785 4879 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:13:20.738926 4879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 08:13:20.739002 4879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:13:20.739094 4879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 08:13:20.739159 4879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 08:13:20.748569 4879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:37541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.21.84.190:46613
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.84.190:46613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.21.84.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:13:20.750051 4879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:13:20.751920 4879 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:13:20.763747 4885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:20.764367 4884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:20.764804 4887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:13:20.765511 4879 server_base.cc:1034] running on GCE node
I20240501 08:13:20.766379 4879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:13:20.767882 4879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:13:20.769094 4879 hybrid_clock.cc:648] HybridClock initialized: now 1714551200769050 us; error 44 us; skew 500 ppm
I20240501 08:13:20.769454 4879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:13:20.771917 4879 webserver.cc:460] Webserver started at http://127.21.84.190:38121/ using document root <none> and password file <none>
I20240501 08:13:20.772675 4879 fs_manager.cc:359] Metadata directory not provided
I20240501 08:13:20.772800 4879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:13:20.773130 4879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:13:20.775900 4879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "3f2a66a8caf94a63a106ecbd26804452"
format_stamp: "Formatted at 2024-05-01 08:13:20 on dist-test-slave-9cgk"
I20240501 08:13:20.776719 4879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "3f2a66a8caf94a63a106ecbd26804452"
format_stamp: "Formatted at 2024-05-01 08:13:20 on dist-test-slave-9cgk"
I20240501 08:13:20.782195 4879 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20240501 08:13:20.785709 4893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:20.787190 4879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:13:20.787389 4879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3f2a66a8caf94a63a106ecbd26804452"
format_stamp: "Formatted at 2024-05-01 08:13:20 on dist-test-slave-9cgk"
I20240501 08:13:20.787611 4879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:13:20.838863 4879 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:13:20.839787 4879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:13:20.840090 4879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:13:20.858556 4879 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.190:46613
I20240501 08:13:20.858579 4944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.190:46613 every 8 connection(s)
I20240501 08:13:20.859998 4879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 08:13:20.863732 4945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:20.869000 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 4879
I20240501 08:13:20.869242 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 08:13:20.874791 4945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f2a66a8caf94a63a106ecbd26804452: Bootstrap starting.
I20240501 08:13:20.877892 4945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f2a66a8caf94a63a106ecbd26804452: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:20.878988 4945 log.cc:824] T 00000000000000000000000000000000 P 3f2a66a8caf94a63a106ecbd26804452: Log is configured to *not* fsync() on all Append() calls
I20240501 08:13:20.882335 4945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f2a66a8caf94a63a106ecbd26804452: No bootstrap required, opened a new log
I20240501 08:13:20.889184 4945 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f2a66a8caf94a63a106ecbd26804452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f2a66a8caf94a63a106ecbd26804452" member_type: VOTER last_known_addr { host: "127.21.84.190" port: 46613 } }
I20240501 08:13:20.889607 4945 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f2a66a8caf94a63a106ecbd26804452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:20.889750 4945 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f2a66a8caf94a63a106ecbd26804452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f2a66a8caf94a63a106ecbd26804452, State: Initialized, Role: FOLLOWER
I20240501 08:13:20.890465 4945 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f2a66a8caf94a63a106ecbd26804452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f2a66a8caf94a63a106ecbd26804452" member_type: VOTER last_known_addr { host: "127.21.84.190" port: 46613 } }
I20240501 08:13:20.890728 4945 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f2a66a8caf94a63a106ecbd26804452 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:13:20.890851 4945 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f2a66a8caf94a63a106ecbd26804452 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:13:20.890967 4945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f2a66a8caf94a63a106ecbd26804452 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:20.892982 4945 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f2a66a8caf94a63a106ecbd26804452 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f2a66a8caf94a63a106ecbd26804452" member_type: VOTER last_known_addr { host: "127.21.84.190" port: 46613 } }
I20240501 08:13:20.893477 4945 leader_election.cc:302] T 00000000000000000000000000000000 P 3f2a66a8caf94a63a106ecbd26804452 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f2a66a8caf94a63a106ecbd26804452; no voters:
I20240501 08:13:20.894152 4945 leader_election.cc:288] T 00000000000000000000000000000000 P 3f2a66a8caf94a63a106ecbd26804452 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:13:20.894295 4950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f2a66a8caf94a63a106ecbd26804452 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:20.894948 4950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f2a66a8caf94a63a106ecbd26804452 [term 1 LEADER]: Becoming Leader. State: Replica: 3f2a66a8caf94a63a106ecbd26804452, State: Running, Role: LEADER
I20240501 08:13:20.895521 4950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f2a66a8caf94a63a106ecbd26804452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f2a66a8caf94a63a106ecbd26804452" member_type: VOTER last_known_addr { host: "127.21.84.190" port: 46613 } }
I20240501 08:13:20.896313 4945 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f2a66a8caf94a63a106ecbd26804452 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:13:20.898124 4952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f2a66a8caf94a63a106ecbd26804452 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f2a66a8caf94a63a106ecbd26804452. Latest consensus state: current_term: 1 leader_uuid: "3f2a66a8caf94a63a106ecbd26804452" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f2a66a8caf94a63a106ecbd26804452" member_type: VOTER last_known_addr { host: "127.21.84.190" port: 46613 } } }
I20240501 08:13:20.898216 4951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f2a66a8caf94a63a106ecbd26804452 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f2a66a8caf94a63a106ecbd26804452" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f2a66a8caf94a63a106ecbd26804452" member_type: VOTER last_known_addr { host: "127.21.84.190" port: 46613 } } }
I20240501 08:13:20.898607 4952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f2a66a8caf94a63a106ecbd26804452 [sys.catalog]: This master's current role is: LEADER
I20240501 08:13:20.898653 4951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f2a66a8caf94a63a106ecbd26804452 [sys.catalog]: This master's current role is: LEADER
I20240501 08:13:20.899857 4957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:13:20.906087 4957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:13:20.918270 4957 catalog_manager.cc:1347] Generated new cluster ID: 708c8c53f69641bd879c7a91634b7545
I20240501 08:13:20.918437 4957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:13:20.945283 4957 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:13:20.946528 4957 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:13:20.953670 4957 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3f2a66a8caf94a63a106ecbd26804452: Generated new TSK 0
I20240501 08:13:20.954576 4957 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:13:20.968246 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.129:0
--local_ip_for_outbound_sockets=127.21.84.129
--webserver_interface=127.21.84.129
--webserver_port=0
--tserver_master_addrs=127.21.84.190:46613
--builtin_ntp_servers=127.21.84.148:37541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 08:13:21.318578 4969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:21.318902 4969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:13:21.319033 4969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:13:21.326581 4969 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:13:21.326732 4969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:13:21.326900 4969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.129
I20240501 08:13:21.336447 4969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:37541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.21.84.129
--webserver_port=0
--tserver_master_addrs=127.21.84.190:46613
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.129
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:13:21.337846 4969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:13:21.339684 4969 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:13:21.352350 4974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:21.353210 4975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:21.354020 4977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:13:21.354871 4969 server_base.cc:1034] running on GCE node
I20240501 08:13:21.355721 4969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:13:21.356768 4969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:13:21.358008 4969 hybrid_clock.cc:648] HybridClock initialized: now 1714551201357888 us; error 113 us; skew 500 ppm
I20240501 08:13:21.358350 4969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:13:21.366163 4969 webserver.cc:460] Webserver started at http://127.21.84.129:38821/ using document root <none> and password file <none>
I20240501 08:13:21.367015 4969 fs_manager.cc:359] Metadata directory not provided
I20240501 08:13:21.367146 4969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:13:21.367537 4969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:13:21.370461 4969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b6cc443a494e4595b0e7cf1777620a3f"
format_stamp: "Formatted at 2024-05-01 08:13:21 on dist-test-slave-9cgk"
I20240501 08:13:21.371289 4969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b6cc443a494e4595b0e7cf1777620a3f"
format_stamp: "Formatted at 2024-05-01 08:13:21 on dist-test-slave-9cgk"
I20240501 08:13:21.376901 4969 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 08:13:21.380712 4983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:21.382213 4969 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:13:21.382393 4969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b6cc443a494e4595b0e7cf1777620a3f"
format_stamp: "Formatted at 2024-05-01 08:13:21 on dist-test-slave-9cgk"
I20240501 08:13:21.382603 4969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:13:21.396183 4969 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:13:21.397163 4969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:13:21.397482 4969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:13:21.398667 4969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:13:21.400211 4969 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:13:21.400301 4969 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:21.400384 4969 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:13:21.400435 4969 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:21.433534 4969 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.129:35417
I20240501 08:13:21.433586 5095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.129:35417 every 8 connection(s)
I20240501 08:13:21.434901 4969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 08:13:21.438994 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 4969
I20240501 08:13:21.439218 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 08:13:21.442711 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.130:0
--local_ip_for_outbound_sockets=127.21.84.130
--webserver_interface=127.21.84.130
--webserver_port=0
--tserver_master_addrs=127.21.84.190:46613
--builtin_ntp_servers=127.21.84.148:37541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:13:21.448654 5096 heartbeater.cc:347] Connected to a master server at 127.21.84.190:46613
I20240501 08:13:21.448987 5096 heartbeater.cc:464] Registering TS with master...
I20240501 08:13:21.449903 5096 heartbeater.cc:511] Master 127.21.84.190:46613 requested a full tablet report, sending...
I20240501 08:13:21.452116 4910 ts_manager.cc:194] Registered new tserver with Master: b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417)
I20240501 08:13:21.454083 4910 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.129:58131
W20240501 08:13:21.783733 5100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:21.784058 5100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:13:21.784204 5100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:13:21.792101 5100 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:13:21.792264 5100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:13:21.792476 5100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.130
I20240501 08:13:21.801883 5100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:37541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.21.84.130
--webserver_port=0
--tserver_master_addrs=127.21.84.190:46613
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.130
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:13:21.803376 5100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:13:21.805171 5100 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:13:21.817874 5105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:21.818506 5106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:21.819034 5108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:13:21.820158 5100 server_base.cc:1034] running on GCE node
I20240501 08:13:21.821059 5100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:13:21.822121 5100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:13:21.823330 5100 hybrid_clock.cc:648] HybridClock initialized: now 1714551201823278 us; error 51 us; skew 500 ppm
I20240501 08:13:21.823645 5100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:13:21.825925 5100 webserver.cc:460] Webserver started at http://127.21.84.130:44865/ using document root <none> and password file <none>
I20240501 08:13:21.826697 5100 fs_manager.cc:359] Metadata directory not provided
I20240501 08:13:21.826814 5100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:13:21.827124 5100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:13:21.830363 5100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
format_stamp: "Formatted at 2024-05-01 08:13:21 on dist-test-slave-9cgk"
I20240501 08:13:21.831241 5100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
format_stamp: "Formatted at 2024-05-01 08:13:21 on dist-test-slave-9cgk"
I20240501 08:13:21.836571 5100 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20240501 08:13:21.840083 5114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:21.841368 5100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 08:13:21.841540 5100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
format_stamp: "Formatted at 2024-05-01 08:13:21 on dist-test-slave-9cgk"
I20240501 08:13:21.841738 5100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:13:21.874998 5100 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:13:21.875895 5100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:13:21.876181 5100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:13:21.877207 5100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:13:21.878778 5100 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:13:21.878878 5100 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:21.878995 5100 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:13:21.879067 5100 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:21.910984 5100 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.130:41937
I20240501 08:13:21.911054 5226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.130:41937 every 8 connection(s)
I20240501 08:13:21.912318 5100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 08:13:21.921087 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 5100
I20240501 08:13:21.921277 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 08:13:21.924461 5227 heartbeater.cc:347] Connected to a master server at 127.21.84.190:46613
I20240501 08:13:21.924538 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.131:0
--local_ip_for_outbound_sockets=127.21.84.131
--webserver_interface=127.21.84.131
--webserver_port=0
--tserver_master_addrs=127.21.84.190:46613
--builtin_ntp_servers=127.21.84.148:37541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:13:21.924772 5227 heartbeater.cc:464] Registering TS with master...
I20240501 08:13:21.925555 5227 heartbeater.cc:511] Master 127.21.84.190:46613 requested a full tablet report, sending...
I20240501 08:13:21.927021 4910 ts_manager.cc:194] Registered new tserver with Master: 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937)
I20240501 08:13:21.928216 4910 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.130:55181
W20240501 08:13:22.265561 5230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:22.265873 5230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:13:22.266010 5230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:13:22.273728 5230 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:13:22.273883 5230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:13:22.274055 5230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.131
I20240501 08:13:22.283389 5230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:37541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.21.84.131
--webserver_port=0
--tserver_master_addrs=127.21.84.190:46613
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.131
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:13:22.284723 5230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:13:22.286530 5230 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:13:22.299050 5235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:22.299706 5238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:22.300447 5236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:13:22.301146 5230 server_base.cc:1034] running on GCE node
I20240501 08:13:22.301755 5230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:13:22.302774 5230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:13:22.304019 5230 hybrid_clock.cc:648] HybridClock initialized: now 1714551202303966 us; error 49 us; skew 500 ppm
I20240501 08:13:22.304361 5230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:13:22.306777 5230 webserver.cc:460] Webserver started at http://127.21.84.131:37807/ using document root <none> and password file <none>
I20240501 08:13:22.307626 5230 fs_manager.cc:359] Metadata directory not provided
I20240501 08:13:22.307754 5230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:13:22.308100 5230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:13:22.310982 5230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "9808124b81e14d1290ab0a6641a4b836"
format_stamp: "Formatted at 2024-05-01 08:13:22 on dist-test-slave-9cgk"
I20240501 08:13:22.311822 5230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "9808124b81e14d1290ab0a6641a4b836"
format_stamp: "Formatted at 2024-05-01 08:13:22 on dist-test-slave-9cgk"
I20240501 08:13:22.317394 5230 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 08:13:22.321133 5244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:22.322535 5230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:13:22.322697 5230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9808124b81e14d1290ab0a6641a4b836"
format_stamp: "Formatted at 2024-05-01 08:13:22 on dist-test-slave-9cgk"
I20240501 08:13:22.322898 5230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:13:22.343120 5230 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:13:22.343997 5230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:13:22.344276 5230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:13:22.345261 5230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:13:22.346938 5230 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:13:22.347038 5230 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:22.347128 5230 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:13:22.347185 5230 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:22.379434 5230 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.131:36983
I20240501 08:13:22.379482 5356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.131:36983 every 8 connection(s)
I20240501 08:13:22.380762 5230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 08:13:22.382478 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 5230
I20240501 08:13:22.382684 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 08:13:22.385711 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.132:0
--local_ip_for_outbound_sockets=127.21.84.132
--webserver_interface=127.21.84.132
--webserver_port=0
--tserver_master_addrs=127.21.84.190:46613
--builtin_ntp_servers=127.21.84.148:37541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:13:22.394551 5357 heartbeater.cc:347] Connected to a master server at 127.21.84.190:46613
I20240501 08:13:22.394877 5357 heartbeater.cc:464] Registering TS with master...
I20240501 08:13:22.395696 5357 heartbeater.cc:511] Master 127.21.84.190:46613 requested a full tablet report, sending...
I20240501 08:13:22.397131 4910 ts_manager.cc:194] Registered new tserver with Master: 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983)
I20240501 08:13:22.398011 4910 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.131:41797
I20240501 08:13:22.457224 5096 heartbeater.cc:503] Master 127.21.84.190:46613 was elected leader, sending a full tablet report...
W20240501 08:13:22.718699 5360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:22.719028 5360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:13:22.719180 5360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:13:22.726395 5360 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:13:22.726549 5360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:13:22.726706 5360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.132
I20240501 08:13:22.735581 5360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:37541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.21.84.132
--webserver_port=0
--tserver_master_addrs=127.21.84.190:46613
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.132
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:13:22.736971 5360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:13:22.738731 5360 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:13:22.751169 5368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:22.751770 5365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:22.752449 5366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:13:22.753538 5360 server_base.cc:1034] running on GCE node
I20240501 08:13:22.754202 5360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:13:22.755177 5360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:13:22.756379 5360 hybrid_clock.cc:648] HybridClock initialized: now 1714551202756337 us; error 37 us; skew 500 ppm
I20240501 08:13:22.756695 5360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:13:22.758947 5360 webserver.cc:460] Webserver started at http://127.21.84.132:45377/ using document root <none> and password file <none>
I20240501 08:13:22.759690 5360 fs_manager.cc:359] Metadata directory not provided
I20240501 08:13:22.759824 5360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:13:22.760149 5360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:13:22.762655 5360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "2c4d718738d1408eabce489f4e140af6"
format_stamp: "Formatted at 2024-05-01 08:13:22 on dist-test-slave-9cgk"
I20240501 08:13:22.763377 5360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "2c4d718738d1408eabce489f4e140af6"
format_stamp: "Formatted at 2024-05-01 08:13:22 on dist-test-slave-9cgk"
I20240501 08:13:22.768504 5360 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 08:13:22.771921 5374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:22.773231 5360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240501 08:13:22.773376 5360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "2c4d718738d1408eabce489f4e140af6"
format_stamp: "Formatted at 2024-05-01 08:13:22 on dist-test-slave-9cgk"
I20240501 08:13:22.773568 5360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:13:22.822149 5360 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:13:22.823015 5360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:13:22.823295 5360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:13:22.824244 5360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:13:22.825727 5360 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:13:22.825826 5360 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:22.825963 5360 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:13:22.826043 5360 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20240501 08:13:22.856889 5360 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.132:34797
I20240501 08:13:22.856942 5486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.132:34797 every 8 connection(s)
I20240501 08:13:22.858363 5360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 08:13:22.863438 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 5360
I20240501 08:13:22.863664 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 08:13:22.867002 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.133:0
--local_ip_for_outbound_sockets=127.21.84.133
--webserver_interface=127.21.84.133
--webserver_port=0
--tserver_master_addrs=127.21.84.190:46613
--builtin_ntp_servers=127.21.84.148:37541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:13:22.871815 5487 heartbeater.cc:347] Connected to a master server at 127.21.84.190:46613
I20240501 08:13:22.872135 5487 heartbeater.cc:464] Registering TS with master...
I20240501 08:13:22.872932 5487 heartbeater.cc:511] Master 127.21.84.190:46613 requested a full tablet report, sending...
I20240501 08:13:22.874461 4910 ts_manager.cc:194] Registered new tserver with Master: 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797)
I20240501 08:13:22.875625 4910 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.132:39595
I20240501 08:13:22.930783 5227 heartbeater.cc:503] Master 127.21.84.190:46613 was elected leader, sending a full tablet report...
W20240501 08:13:23.202859 5490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:23.203182 5490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:13:23.203349 5490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:13:23.210707 5490 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:13:23.210841 5490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:13:23.210956 5490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.133
I20240501 08:13:23.220072 5490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:37541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.21.84.133
--webserver_port=0
--tserver_master_addrs=127.21.84.190:46613
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.133
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:13:23.221413 5490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:13:23.223210 5490 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:13:23.235743 5495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:23.236411 5496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:23.237318 5498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:13:23.237823 5490 server_base.cc:1034] running on GCE node
I20240501 08:13:23.238467 5490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:13:23.239496 5490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:13:23.240731 5490 hybrid_clock.cc:648] HybridClock initialized: now 1714551203240660 us; error 57 us; skew 500 ppm
I20240501 08:13:23.241073 5490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:13:23.243402 5490 webserver.cc:460] Webserver started at http://127.21.84.133:40227/ using document root <none> and password file <none>
I20240501 08:13:23.244199 5490 fs_manager.cc:359] Metadata directory not provided
I20240501 08:13:23.244321 5490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:13:23.244642 5490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:13:23.247424 5490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "6651d7356f5c48388f38c1034d466962"
format_stamp: "Formatted at 2024-05-01 08:13:23 on dist-test-slave-9cgk"
I20240501 08:13:23.248184 5490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "6651d7356f5c48388f38c1034d466962"
format_stamp: "Formatted at 2024-05-01 08:13:23 on dist-test-slave-9cgk"
I20240501 08:13:23.253541 5490 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 08:13:23.257140 5504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:23.258647 5490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:13:23.258846 5490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "6651d7356f5c48388f38c1034d466962"
format_stamp: "Formatted at 2024-05-01 08:13:23 on dist-test-slave-9cgk"
I20240501 08:13:23.259078 5490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:13:23.280506 5490 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:13:23.281389 5490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:13:23.281649 5490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:13:23.282666 5490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:13:23.284168 5490 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:13:23.284255 5490 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:23.284368 5490 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:13:23.284436 5490 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:23.315891 5490 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.133:32805
I20240501 08:13:23.315937 5616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.133:32805 every 8 connection(s)
I20240501 08:13:23.317210 5490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 08:13:23.325918 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 5490
I20240501 08:13:23.326153 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 08:13:23.329707 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.134:0
--local_ip_for_outbound_sockets=127.21.84.134
--webserver_interface=127.21.84.134
--webserver_port=0
--tserver_master_addrs=127.21.84.190:46613
--builtin_ntp_servers=127.21.84.148:37541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:13:23.330658 5617 heartbeater.cc:347] Connected to a master server at 127.21.84.190:46613
I20240501 08:13:23.330963 5617 heartbeater.cc:464] Registering TS with master...
I20240501 08:13:23.331794 5617 heartbeater.cc:511] Master 127.21.84.190:46613 requested a full tablet report, sending...
I20240501 08:13:23.333140 4910 ts_manager.cc:194] Registered new tserver with Master: 6651d7356f5c48388f38c1034d466962 (127.21.84.133:32805)
I20240501 08:13:23.333897 4910 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.133:59193
I20240501 08:13:23.400569 5357 heartbeater.cc:503] Master 127.21.84.190:46613 was elected leader, sending a full tablet report...
W20240501 08:13:23.670809 5620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:23.671168 5620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:13:23.671316 5620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:13:23.678835 5620 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:13:23.679003 5620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:13:23.679159 5620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.134
I20240501 08:13:23.688450 5620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:37541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.21.84.134
--webserver_port=0
--tserver_master_addrs=127.21.84.190:46613
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.134
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:13:23.689901 5620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:13:23.691679 5620 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:13:23.703737 5625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:23.704526 5628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:23.705250 5626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:13:23.706122 5620 server_base.cc:1034] running on GCE node
I20240501 08:13:23.706765 5620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:13:23.707686 5620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:13:23.708886 5620 hybrid_clock.cc:648] HybridClock initialized: now 1714551203708830 us; error 44 us; skew 500 ppm
I20240501 08:13:23.709206 5620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:13:23.711530 5620 webserver.cc:460] Webserver started at http://127.21.84.134:36543/ using document root <none> and password file <none>
I20240501 08:13:23.712267 5620 fs_manager.cc:359] Metadata directory not provided
I20240501 08:13:23.712381 5620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:13:23.712692 5620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:13:23.715567 5620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "36c507d2fe0c4ede9556db4ab9b563b1"
format_stamp: "Formatted at 2024-05-01 08:13:23 on dist-test-slave-9cgk"
I20240501 08:13:23.716373 5620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "36c507d2fe0c4ede9556db4ab9b563b1"
format_stamp: "Formatted at 2024-05-01 08:13:23 on dist-test-slave-9cgk"
I20240501 08:13:23.721714 5620 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 08:13:23.725219 5634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:23.726650 5620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:13:23.726815 5620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "36c507d2fe0c4ede9556db4ab9b563b1"
format_stamp: "Formatted at 2024-05-01 08:13:23 on dist-test-slave-9cgk"
I20240501 08:13:23.727052 5620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:13:23.744513 5620 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:13:23.745417 5620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:13:23.745757 5620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:13:23.746847 5620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:13:23.748404 5620 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:13:23.748500 5620 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:23.748620 5620 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:13:23.748696 5620 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:23.781872 5620 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.134:39137
I20240501 08:13:23.781914 5746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.134:39137 every 8 connection(s)
I20240501 08:13:23.783244 5620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 08:13:23.786758 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 5620
I20240501 08:13:23.786950 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 08:13:23.790634 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.135:0
--local_ip_for_outbound_sockets=127.21.84.135
--webserver_interface=127.21.84.135
--webserver_port=0
--tserver_master_addrs=127.21.84.190:46613
--builtin_ntp_servers=127.21.84.148:37541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:13:23.797466 5747 heartbeater.cc:347] Connected to a master server at 127.21.84.190:46613
I20240501 08:13:23.797775 5747 heartbeater.cc:464] Registering TS with master...
I20240501 08:13:23.798656 5747 heartbeater.cc:511] Master 127.21.84.190:46613 requested a full tablet report, sending...
I20240501 08:13:23.800165 4910 ts_manager.cc:194] Registered new tserver with Master: 36c507d2fe0c4ede9556db4ab9b563b1 (127.21.84.134:39137)
I20240501 08:13:23.800936 4910 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.134:35003
I20240501 08:13:23.878835 5487 heartbeater.cc:503] Master 127.21.84.190:46613 was elected leader, sending a full tablet report...
W20240501 08:13:24.133181 5750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:24.133527 5750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:13:24.133672 5750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:13:24.141058 5750 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:13:24.141220 5750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:13:24.141448 5750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.135
I20240501 08:13:24.151355 5750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:37541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.21.84.135
--webserver_port=0
--tserver_master_addrs=127.21.84.190:46613
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.135
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:13:24.152673 5750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:13:24.154403 5750 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:13:24.167165 5756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:24.167696 5758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:24.168474 5755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:13:24.169464 5750 server_base.cc:1034] running on GCE node
I20240501 08:13:24.170132 5750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:13:24.171124 5750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:13:24.172326 5750 hybrid_clock.cc:648] HybridClock initialized: now 1714551204172262 us; error 56 us; skew 500 ppm
I20240501 08:13:24.172610 5750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:13:24.174908 5750 webserver.cc:460] Webserver started at http://127.21.84.135:46769/ using document root <none> and password file <none>
I20240501 08:13:24.175644 5750 fs_manager.cc:359] Metadata directory not provided
I20240501 08:13:24.175747 5750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:13:24.176028 5750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:13:24.178876 5750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "eb21f6475035413984756905636d034f"
format_stamp: "Formatted at 2024-05-01 08:13:24 on dist-test-slave-9cgk"
I20240501 08:13:24.179690 5750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "eb21f6475035413984756905636d034f"
format_stamp: "Formatted at 2024-05-01 08:13:24 on dist-test-slave-9cgk"
I20240501 08:13:24.185276 5750 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20240501 08:13:24.189039 5764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:24.190519 5750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240501 08:13:24.190694 5750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "eb21f6475035413984756905636d034f"
format_stamp: "Formatted at 2024-05-01 08:13:24 on dist-test-slave-9cgk"
I20240501 08:13:24.190915 5750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:13:24.218999 5750 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:13:24.219903 5750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:13:24.220202 5750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:13:24.221184 5750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:13:24.222744 5750 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:13:24.222846 5750 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:24.222962 5750 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:13:24.223039 5750 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:24.252243 5750 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.135:44223
I20240501 08:13:24.252339 5876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.135:44223 every 8 connection(s)
I20240501 08:13:24.253674 5750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 08:13:24.259711 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 5750
I20240501 08:13:24.259940 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 08:13:24.264333 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.136:0
--local_ip_for_outbound_sockets=127.21.84.136
--webserver_interface=127.21.84.136
--webserver_port=0
--tserver_master_addrs=127.21.84.190:46613
--builtin_ntp_servers=127.21.84.148:37541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:13:24.276932 5877 heartbeater.cc:347] Connected to a master server at 127.21.84.190:46613
I20240501 08:13:24.277321 5877 heartbeater.cc:464] Registering TS with master...
I20240501 08:13:24.278298 5877 heartbeater.cc:511] Master 127.21.84.190:46613 requested a full tablet report, sending...
I20240501 08:13:24.279883 4910 ts_manager.cc:194] Registered new tserver with Master: eb21f6475035413984756905636d034f (127.21.84.135:44223)
I20240501 08:13:24.280700 4910 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.135:48105
I20240501 08:13:24.336385 5617 heartbeater.cc:503] Master 127.21.84.190:46613 was elected leader, sending a full tablet report...
W20240501 08:13:24.705241 5880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:24.705662 5880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:13:24.705819 5880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:13:24.713109 5880 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:13:24.713228 5880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:13:24.713378 5880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.136
I20240501 08:13:24.722035 5880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:37541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.21.84.136
--webserver_port=0
--tserver_master_addrs=127.21.84.190:46613
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.136
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:13:24.723346 5880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:13:24.724953 5880 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:13:24.737156 5886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:24.737818 5885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:24.738715 5888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:13:24.739631 5880 server_base.cc:1034] running on GCE node
I20240501 08:13:24.740250 5880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:13:24.741191 5880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:13:24.742393 5880 hybrid_clock.cc:648] HybridClock initialized: now 1714551204742321 us; error 53 us; skew 500 ppm
I20240501 08:13:24.742694 5880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:13:24.744912 5880 webserver.cc:460] Webserver started at http://127.21.84.136:43427/ using document root <none> and password file <none>
I20240501 08:13:24.745645 5880 fs_manager.cc:359] Metadata directory not provided
I20240501 08:13:24.745755 5880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:13:24.746120 5880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:13:24.748690 5880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "67b454b1f4f344ebbfe96dced538e368"
format_stamp: "Formatted at 2024-05-01 08:13:24 on dist-test-slave-9cgk"
I20240501 08:13:24.749518 5880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "67b454b1f4f344ebbfe96dced538e368"
format_stamp: "Formatted at 2024-05-01 08:13:24 on dist-test-slave-9cgk"
I20240501 08:13:24.755148 5880 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 08:13:24.758692 5894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:24.760048 5880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 08:13:24.760229 5880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "67b454b1f4f344ebbfe96dced538e368"
format_stamp: "Formatted at 2024-05-01 08:13:24 on dist-test-slave-9cgk"
I20240501 08:13:24.760421 5880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:13:24.782459 5880 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:13:24.783272 5880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:13:24.783522 5880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:13:24.784477 5880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:13:24.785887 5880 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:13:24.786016 5880 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:24.786114 5880 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:13:24.786175 5880 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:24.803599 5747 heartbeater.cc:503] Master 127.21.84.190:46613 was elected leader, sending a full tablet report...
I20240501 08:13:24.816762 5880 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.136:37697
I20240501 08:13:24.816809 6006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.136:37697 every 8 connection(s)
I20240501 08:13:24.818113 5880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 08:13:24.823365 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 5880
I20240501 08:13:24.823518 21842 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 08:13:24.830765 6007 heartbeater.cc:347] Connected to a master server at 127.21.84.190:46613
I20240501 08:13:24.831061 6007 heartbeater.cc:464] Registering TS with master...
I20240501 08:13:24.831894 6007 heartbeater.cc:511] Master 127.21.84.190:46613 requested a full tablet report, sending...
I20240501 08:13:24.833257 4910 ts_manager.cc:194] Registered new tserver with Master: 67b454b1f4f344ebbfe96dced538e368 (127.21.84.136:37697)
I20240501 08:13:24.834161 4910 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.136:36653
I20240501 08:13:24.839196 21842 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 08:13:24.855211 21842 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 08:13:24.855340 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 5490
I20240501 08:13:24.864276 5612 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:13:24.942351 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 5490
I20240501 08:13:24.959287 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 5620
I20240501 08:13:24.968020 5742 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:13:25.046061 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 5620
I20240501 08:13:25.061880 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 5750
I20240501 08:13:25.069988 5872 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:13:25.154165 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 5750
I20240501 08:13:25.170641 21842 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu pid 5880
I20240501 08:13:25.180941 6002 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:13:25.259424 21842 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu with pid 5880
I20240501 08:13:29.036222 4910 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33308:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:13:29.084131 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 416ef8d814614a7caba4279e64893b27 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d0bcba50fe614d74ba984394fb983868]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:29.085366 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 8a96986cf35c4720991739af0626e1e6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d0bcba50fe614d74ba984394fb983868]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:29.086668 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a96986cf35c4720991739af0626e1e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:29.087158 5031 tablet_service.cc:1467] Processing CreateTablet for tablet 416ef8d814614a7caba4279e64893b27 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d0bcba50fe614d74ba984394fb983868]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:29.088702 5031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 416ef8d814614a7caba4279e64893b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:29.089367 5029 tablet_service.cc:1467] Processing CreateTablet for tablet 8a96986cf35c4720991739af0626e1e6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d0bcba50fe614d74ba984394fb983868]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:29.089912 5029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a96986cf35c4720991739af0626e1e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:29.090399 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 416ef8d814614a7caba4279e64893b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:29.090762 5030 tablet_service.cc:1467] Processing CreateTablet for tablet 3fa99ffab5a44c588738be4476ed0fdc (DEFAULT_TABLE table=rebalance_test_table_0 [id=d0bcba50fe614d74ba984394fb983868]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:29.091315 5030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fa99ffab5a44c588738be4476ed0fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:29.097774 5161 tablet_service.cc:1467] Processing CreateTablet for tablet 3fa99ffab5a44c588738be4476ed0fdc (DEFAULT_TABLE table=rebalance_test_table_0 [id=d0bcba50fe614d74ba984394fb983868]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:29.097807 5162 tablet_service.cc:1467] Processing CreateTablet for tablet 416ef8d814614a7caba4279e64893b27 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d0bcba50fe614d74ba984394fb983868]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:29.098476 5291 tablet_service.cc:1467] Processing CreateTablet for tablet 8a96986cf35c4720991739af0626e1e6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d0bcba50fe614d74ba984394fb983868]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:29.099529 5161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fa99ffab5a44c588738be4476ed0fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:29.099785 5291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a96986cf35c4720991739af0626e1e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:29.100049 5162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 416ef8d814614a7caba4279e64893b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:29.097746 5292 tablet_service.cc:1467] Processing CreateTablet for tablet 3fa99ffab5a44c588738be4476ed0fdc (DEFAULT_TABLE table=rebalance_test_table_0 [id=d0bcba50fe614d74ba984394fb983868]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:29.102608 5292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fa99ffab5a44c588738be4476ed0fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:29.140170 6037 tablet_bootstrap.cc:492] T 3fa99ffab5a44c588738be4476ed0fdc P 0a952997a9fc4bfb8d7cbc56de6e0598: Bootstrap starting.
I20240501 08:13:29.140950 6038 tablet_bootstrap.cc:492] T 3fa99ffab5a44c588738be4476ed0fdc P 9808124b81e14d1290ab0a6641a4b836: Bootstrap starting.
I20240501 08:13:29.141150 6036 tablet_bootstrap.cc:492] T 8a96986cf35c4720991739af0626e1e6 P b6cc443a494e4595b0e7cf1777620a3f: Bootstrap starting.
I20240501 08:13:29.143445 6039 tablet_bootstrap.cc:492] T 416ef8d814614a7caba4279e64893b27 P 2c4d718738d1408eabce489f4e140af6: Bootstrap starting.
I20240501 08:13:29.144124 6037 tablet_bootstrap.cc:654] T 3fa99ffab5a44c588738be4476ed0fdc P 0a952997a9fc4bfb8d7cbc56de6e0598: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:29.145033 6036 tablet_bootstrap.cc:654] T 8a96986cf35c4720991739af0626e1e6 P b6cc443a494e4595b0e7cf1777620a3f: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:29.145500 6037 log.cc:824] T 3fa99ffab5a44c588738be4476ed0fdc P 0a952997a9fc4bfb8d7cbc56de6e0598: Log is configured to *not* fsync() on all Append() calls
I20240501 08:13:29.146126 6039 tablet_bootstrap.cc:654] T 416ef8d814614a7caba4279e64893b27 P 2c4d718738d1408eabce489f4e140af6: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:29.146481 6036 log.cc:824] T 8a96986cf35c4720991739af0626e1e6 P b6cc443a494e4595b0e7cf1777620a3f: Log is configured to *not* fsync() on all Append() calls
I20240501 08:13:29.148422 6039 log.cc:824] T 416ef8d814614a7caba4279e64893b27 P 2c4d718738d1408eabce489f4e140af6: Log is configured to *not* fsync() on all Append() calls
I20240501 08:13:29.148607 6038 tablet_bootstrap.cc:654] T 3fa99ffab5a44c588738be4476ed0fdc P 9808124b81e14d1290ab0a6641a4b836: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:29.150171 6038 log.cc:824] T 3fa99ffab5a44c588738be4476ed0fdc P 9808124b81e14d1290ab0a6641a4b836: Log is configured to *not* fsync() on all Append() calls
I20240501 08:13:29.154234 6036 tablet_bootstrap.cc:492] T 8a96986cf35c4720991739af0626e1e6 P b6cc443a494e4595b0e7cf1777620a3f: No bootstrap required, opened a new log
I20240501 08:13:29.154759 6036 ts_tablet_manager.cc:1367] T 8a96986cf35c4720991739af0626e1e6 P b6cc443a494e4595b0e7cf1777620a3f: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.003s
I20240501 08:13:29.160231 6039 tablet_bootstrap.cc:492] T 416ef8d814614a7caba4279e64893b27 P 2c4d718738d1408eabce489f4e140af6: No bootstrap required, opened a new log
I20240501 08:13:29.160835 6039 ts_tablet_manager.cc:1367] T 416ef8d814614a7caba4279e64893b27 P 2c4d718738d1408eabce489f4e140af6: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.010s
I20240501 08:13:29.162431 6037 tablet_bootstrap.cc:492] T 3fa99ffab5a44c588738be4476ed0fdc P 0a952997a9fc4bfb8d7cbc56de6e0598: No bootstrap required, opened a new log
I20240501 08:13:29.162806 6037 ts_tablet_manager.cc:1367] T 3fa99ffab5a44c588738be4476ed0fdc P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.012s
I20240501 08:13:29.164626 6038 tablet_bootstrap.cc:492] T 3fa99ffab5a44c588738be4476ed0fdc P 9808124b81e14d1290ab0a6641a4b836: No bootstrap required, opened a new log
I20240501 08:13:29.164994 6038 ts_tablet_manager.cc:1367] T 3fa99ffab5a44c588738be4476ed0fdc P 9808124b81e14d1290ab0a6641a4b836: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.012s
I20240501 08:13:29.167596 6036 raft_consensus.cc:348] T 8a96986cf35c4720991739af0626e1e6 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:29.168177 6036 raft_consensus.cc:374] T 8a96986cf35c4720991739af0626e1e6 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:29.168327 6036 raft_consensus.cc:729] T 8a96986cf35c4720991739af0626e1e6 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6cc443a494e4595b0e7cf1777620a3f, State: Initialized, Role: FOLLOWER
I20240501 08:13:29.169173 6036 consensus_queue.cc:260] T 8a96986cf35c4720991739af0626e1e6 P b6cc443a494e4595b0e7cf1777620a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:29.171840 6039 raft_consensus.cc:348] T 416ef8d814614a7caba4279e64893b27 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:29.172569 6039 raft_consensus.cc:374] T 416ef8d814614a7caba4279e64893b27 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:29.172726 6039 raft_consensus.cc:729] T 416ef8d814614a7caba4279e64893b27 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Initialized, Role: FOLLOWER
I20240501 08:13:29.173661 6039 consensus_queue.cc:260] T 416ef8d814614a7caba4279e64893b27 P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:29.173975 6037 raft_consensus.cc:348] T 3fa99ffab5a44c588738be4476ed0fdc P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:29.174683 6037 raft_consensus.cc:374] T 3fa99ffab5a44c588738be4476ed0fdc P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:29.174862 6037 raft_consensus.cc:729] T 3fa99ffab5a44c588738be4476ed0fdc P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Initialized, Role: FOLLOWER
I20240501 08:13:29.174988 6038 raft_consensus.cc:348] T 3fa99ffab5a44c588738be4476ed0fdc P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:29.175624 6038 raft_consensus.cc:374] T 3fa99ffab5a44c588738be4476ed0fdc P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:29.175794 6038 raft_consensus.cc:729] T 3fa99ffab5a44c588738be4476ed0fdc P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Initialized, Role: FOLLOWER
I20240501 08:13:29.176167 6037 consensus_queue.cc:260] T 3fa99ffab5a44c588738be4476ed0fdc P 0a952997a9fc4bfb8d7cbc56de6e0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:29.178769 6036 ts_tablet_manager.cc:1398] T 8a96986cf35c4720991739af0626e1e6 P b6cc443a494e4595b0e7cf1777620a3f: Time spent starting tablet: real 0.024s user 0.012s sys 0.009s
I20240501 08:13:29.179481 6036 tablet_bootstrap.cc:492] T 3fa99ffab5a44c588738be4476ed0fdc P b6cc443a494e4595b0e7cf1777620a3f: Bootstrap starting.
I20240501 08:13:29.177222 6038 consensus_queue.cc:260] T 3fa99ffab5a44c588738be4476ed0fdc P 9808124b81e14d1290ab0a6641a4b836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:29.181802 6036 tablet_bootstrap.cc:654] T 3fa99ffab5a44c588738be4476ed0fdc P b6cc443a494e4595b0e7cf1777620a3f: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:29.185310 6037 ts_tablet_manager.cc:1398] T 3fa99ffab5a44c588738be4476ed0fdc P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent starting tablet: real 0.022s user 0.015s sys 0.002s
I20240501 08:13:29.185921 6037 tablet_bootstrap.cc:492] T 416ef8d814614a7caba4279e64893b27 P 0a952997a9fc4bfb8d7cbc56de6e0598: Bootstrap starting.
I20240501 08:13:29.188802 6039 ts_tablet_manager.cc:1398] T 416ef8d814614a7caba4279e64893b27 P 2c4d718738d1408eabce489f4e140af6: Time spent starting tablet: real 0.028s user 0.012s sys 0.005s
I20240501 08:13:29.189740 6037 tablet_bootstrap.cc:654] T 416ef8d814614a7caba4279e64893b27 P 0a952997a9fc4bfb8d7cbc56de6e0598: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:29.189777 6039 tablet_bootstrap.cc:492] T 8a96986cf35c4720991739af0626e1e6 P 2c4d718738d1408eabce489f4e140af6: Bootstrap starting.
W20240501 08:13:29.192024 5097 tablet.cc:2369] T 8a96986cf35c4720991739af0626e1e6 P b6cc443a494e4595b0e7cf1777620a3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:13:29.194150 6039 tablet_bootstrap.cc:654] T 8a96986cf35c4720991739af0626e1e6 P 2c4d718738d1408eabce489f4e140af6: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:29.195063 6036 tablet_bootstrap.cc:492] T 3fa99ffab5a44c588738be4476ed0fdc P b6cc443a494e4595b0e7cf1777620a3f: No bootstrap required, opened a new log
I20240501 08:13:29.195266 6036 ts_tablet_manager.cc:1367] T 3fa99ffab5a44c588738be4476ed0fdc P b6cc443a494e4595b0e7cf1777620a3f: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.003s
I20240501 08:13:29.196146 6037 tablet_bootstrap.cc:492] T 416ef8d814614a7caba4279e64893b27 P 0a952997a9fc4bfb8d7cbc56de6e0598: No bootstrap required, opened a new log
I20240501 08:13:29.196343 6037 ts_tablet_manager.cc:1367] T 416ef8d814614a7caba4279e64893b27 P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240501 08:13:29.196424 6036 raft_consensus.cc:348] T 3fa99ffab5a44c588738be4476ed0fdc P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:29.196663 6036 raft_consensus.cc:374] T 3fa99ffab5a44c588738be4476ed0fdc P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:29.196758 6036 raft_consensus.cc:729] T 3fa99ffab5a44c588738be4476ed0fdc P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6cc443a494e4595b0e7cf1777620a3f, State: Initialized, Role: FOLLOWER
I20240501 08:13:29.197013 6036 consensus_queue.cc:260] T 3fa99ffab5a44c588738be4476ed0fdc P b6cc443a494e4595b0e7cf1777620a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:29.197639 6037 raft_consensus.cc:348] T 416ef8d814614a7caba4279e64893b27 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:29.197881 6037 raft_consensus.cc:374] T 416ef8d814614a7caba4279e64893b27 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:29.198040 6037 raft_consensus.cc:729] T 416ef8d814614a7caba4279e64893b27 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Initialized, Role: FOLLOWER
I20240501 08:13:29.198369 6037 consensus_queue.cc:260] T 416ef8d814614a7caba4279e64893b27 P 0a952997a9fc4bfb8d7cbc56de6e0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:29.198657 6038 ts_tablet_manager.cc:1398] T 3fa99ffab5a44c588738be4476ed0fdc P 9808124b81e14d1290ab0a6641a4b836: Time spent starting tablet: real 0.034s user 0.022s sys 0.005s
I20240501 08:13:29.199345 6038 tablet_bootstrap.cc:492] T 8a96986cf35c4720991739af0626e1e6 P 9808124b81e14d1290ab0a6641a4b836: Bootstrap starting.
I20240501 08:13:29.200347 6046 raft_consensus.cc:482] T 416ef8d814614a7caba4279e64893b27 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:29.200594 6046 raft_consensus.cc:504] T 416ef8d814614a7caba4279e64893b27 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:29.201123 6036 ts_tablet_manager.cc:1398] T 3fa99ffab5a44c588738be4476ed0fdc P b6cc443a494e4595b0e7cf1777620a3f: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20240501 08:13:29.201459 6036 tablet_bootstrap.cc:492] T 416ef8d814614a7caba4279e64893b27 P b6cc443a494e4595b0e7cf1777620a3f: Bootstrap starting.
I20240501 08:13:29.201905 6038 tablet_bootstrap.cc:654] T 8a96986cf35c4720991739af0626e1e6 P 9808124b81e14d1290ab0a6641a4b836: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:29.202780 6046 leader_election.cc:288] T 416ef8d814614a7caba4279e64893b27 P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797), b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417)
I20240501 08:13:29.207255 6039 tablet_bootstrap.cc:492] T 8a96986cf35c4720991739af0626e1e6 P 2c4d718738d1408eabce489f4e140af6: No bootstrap required, opened a new log
I20240501 08:13:29.207625 6039 ts_tablet_manager.cc:1367] T 8a96986cf35c4720991739af0626e1e6 P 2c4d718738d1408eabce489f4e140af6: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.001s
I20240501 08:13:29.208325 6036 tablet_bootstrap.cc:654] T 416ef8d814614a7caba4279e64893b27 P b6cc443a494e4595b0e7cf1777620a3f: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:29.209069 6039 raft_consensus.cc:348] T 8a96986cf35c4720991739af0626e1e6 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:29.209410 6039 raft_consensus.cc:374] T 8a96986cf35c4720991739af0626e1e6 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:29.209589 6039 raft_consensus.cc:729] T 8a96986cf35c4720991739af0626e1e6 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Initialized, Role: FOLLOWER
I20240501 08:13:29.210003 6039 consensus_queue.cc:260] T 8a96986cf35c4720991739af0626e1e6 P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:29.210361 6038 tablet_bootstrap.cc:492] T 8a96986cf35c4720991739af0626e1e6 P 9808124b81e14d1290ab0a6641a4b836: No bootstrap required, opened a new log
I20240501 08:13:29.210568 6038 ts_tablet_manager.cc:1367] T 8a96986cf35c4720991739af0626e1e6 P 9808124b81e14d1290ab0a6641a4b836: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.008s
I20240501 08:13:29.211608 6039 ts_tablet_manager.cc:1398] T 8a96986cf35c4720991739af0626e1e6 P 2c4d718738d1408eabce489f4e140af6: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:13:29.211699 6038 raft_consensus.cc:348] T 8a96986cf35c4720991739af0626e1e6 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:29.211916 6038 raft_consensus.cc:374] T 8a96986cf35c4720991739af0626e1e6 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:29.212023 6038 raft_consensus.cc:729] T 8a96986cf35c4720991739af0626e1e6 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Initialized, Role: FOLLOWER
I20240501 08:13:29.212273 6038 consensus_queue.cc:260] T 8a96986cf35c4720991739af0626e1e6 P 9808124b81e14d1290ab0a6641a4b836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:29.212760 6038 ts_tablet_manager.cc:1398] T 8a96986cf35c4720991739af0626e1e6 P 9808124b81e14d1290ab0a6641a4b836: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:29.212738 6037 ts_tablet_manager.cc:1398] T 416ef8d814614a7caba4279e64893b27 P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240501 08:13:29.220350 6036 tablet_bootstrap.cc:492] T 416ef8d814614a7caba4279e64893b27 P b6cc443a494e4595b0e7cf1777620a3f: No bootstrap required, opened a new log
I20240501 08:13:29.220674 6036 ts_tablet_manager.cc:1367] T 416ef8d814614a7caba4279e64893b27 P b6cc443a494e4595b0e7cf1777620a3f: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.001s
I20240501 08:13:29.221585 6036 raft_consensus.cc:348] T 416ef8d814614a7caba4279e64893b27 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:29.221876 6036 raft_consensus.cc:374] T 416ef8d814614a7caba4279e64893b27 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:29.221990 6036 raft_consensus.cc:729] T 416ef8d814614a7caba4279e64893b27 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6cc443a494e4595b0e7cf1777620a3f, State: Initialized, Role: FOLLOWER
I20240501 08:13:29.222276 6036 consensus_queue.cc:260] T 416ef8d814614a7caba4279e64893b27 P b6cc443a494e4595b0e7cf1777620a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:29.222887 6036 ts_tablet_manager.cc:1398] T 416ef8d814614a7caba4279e64893b27 P b6cc443a494e4595b0e7cf1777620a3f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:13:29.226359 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "416ef8d814614a7caba4279e64893b27" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f" is_pre_election: true
I20240501 08:13:29.226753 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "416ef8d814614a7caba4279e64893b27" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4d718738d1408eabce489f4e140af6" is_pre_election: true
I20240501 08:13:29.227006 5051 raft_consensus.cc:2454] T 416ef8d814614a7caba4279e64893b27 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 0.
I20240501 08:13:29.227270 5442 raft_consensus.cc:2454] T 416ef8d814614a7caba4279e64893b27 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 0.
I20240501 08:13:29.227718 5117 leader_election.cc:302] T 416ef8d814614a7caba4279e64893b27 P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:29.228153 6046 raft_consensus.cc:2781] T 416ef8d814614a7caba4279e64893b27 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:29.228314 6046 raft_consensus.cc:482] T 416ef8d814614a7caba4279e64893b27 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:29.228435 6046 raft_consensus.cc:3037] T 416ef8d814614a7caba4279e64893b27 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:29.230867 6046 raft_consensus.cc:504] T 416ef8d814614a7caba4279e64893b27 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:29.231494 6046 leader_election.cc:288] T 416ef8d814614a7caba4279e64893b27 P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 election: Requested vote from peers 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797), b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417)
I20240501 08:13:29.231994 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "416ef8d814614a7caba4279e64893b27" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:29.232077 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "416ef8d814614a7caba4279e64893b27" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:29.232172 5442 raft_consensus.cc:3037] T 416ef8d814614a7caba4279e64893b27 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:29.232308 5051 raft_consensus.cc:3037] T 416ef8d814614a7caba4279e64893b27 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:29.233367 6045 raft_consensus.cc:482] T 8a96986cf35c4720991739af0626e1e6 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:29.233554 6045 raft_consensus.cc:504] T 8a96986cf35c4720991739af0626e1e6 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:29.234642 5051 raft_consensus.cc:2454] T 416ef8d814614a7caba4279e64893b27 P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 1.
I20240501 08:13:29.234921 5442 raft_consensus.cc:2454] T 416ef8d814614a7caba4279e64893b27 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 1.
I20240501 08:13:29.235086 5117 leader_election.cc:302] T 416ef8d814614a7caba4279e64893b27 P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:29.235415 6046 raft_consensus.cc:2781] T 416ef8d814614a7caba4279e64893b27 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:29.235636 6045 leader_election.cc:288] T 8a96986cf35c4720991739af0626e1e6 P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983), b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417)
I20240501 08:13:29.236296 6046 raft_consensus.cc:686] T 416ef8d814614a7caba4279e64893b27 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Becoming Leader. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Running, Role: LEADER
I20240501 08:13:29.236793 6046 consensus_queue.cc:237] T 416ef8d814614a7caba4279e64893b27 P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:29.243649 5312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a96986cf35c4720991739af0626e1e6" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808124b81e14d1290ab0a6641a4b836" is_pre_election: true
I20240501 08:13:29.244174 5312 raft_consensus.cc:2454] T 8a96986cf35c4720991739af0626e1e6 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 0.
I20240501 08:13:29.243644 4909 catalog_manager.cc:5484] T 416ef8d814614a7caba4279e64893b27 P 0a952997a9fc4bfb8d7cbc56de6e0598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130). New cstate: current_term: 1 leader_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: HEALTHY } } }
I20240501 08:13:29.244557 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a96986cf35c4720991739af0626e1e6" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f" is_pre_election: true
I20240501 08:13:29.244796 5051 raft_consensus.cc:2454] T 8a96986cf35c4720991739af0626e1e6 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 0.
I20240501 08:13:29.244812 5377 leader_election.cc:302] T 8a96986cf35c4720991739af0626e1e6 P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4d718738d1408eabce489f4e140af6, 9808124b81e14d1290ab0a6641a4b836; no voters:
I20240501 08:13:29.245256 6045 raft_consensus.cc:2781] T 8a96986cf35c4720991739af0626e1e6 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:29.245363 6045 raft_consensus.cc:482] T 8a96986cf35c4720991739af0626e1e6 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:29.245460 6045 raft_consensus.cc:3037] T 8a96986cf35c4720991739af0626e1e6 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:29.247849 6045 raft_consensus.cc:504] T 8a96986cf35c4720991739af0626e1e6 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:29.248476 6045 leader_election.cc:288] T 8a96986cf35c4720991739af0626e1e6 P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 1 election: Requested vote from peers 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983), b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417)
I20240501 08:13:29.249414 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a96986cf35c4720991739af0626e1e6" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:29.249640 5051 raft_consensus.cc:3037] T 8a96986cf35c4720991739af0626e1e6 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:29.249576 5312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a96986cf35c4720991739af0626e1e6" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
I20240501 08:13:29.249775 5312 raft_consensus.cc:3037] T 8a96986cf35c4720991739af0626e1e6 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:29.251602 5051 raft_consensus.cc:2454] T 8a96986cf35c4720991739af0626e1e6 P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 1.
I20240501 08:13:29.251997 5312 raft_consensus.cc:2454] T 8a96986cf35c4720991739af0626e1e6 P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 1.
I20240501 08:13:29.252024 5377 leader_election.cc:302] T 8a96986cf35c4720991739af0626e1e6 P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4d718738d1408eabce489f4e140af6, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:29.252389 6045 raft_consensus.cc:2781] T 8a96986cf35c4720991739af0626e1e6 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:29.252811 6045 raft_consensus.cc:686] T 8a96986cf35c4720991739af0626e1e6 P 2c4d718738d1408eabce489f4e140af6 [term 1 LEADER]: Becoming Leader. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:29.253321 6045 consensus_queue.cc:237] T 8a96986cf35c4720991739af0626e1e6 P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:29.257719 4909 catalog_manager.cc:5484] T 8a96986cf35c4720991739af0626e1e6 P 2c4d718738d1408eabce489f4e140af6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c4d718738d1408eabce489f4e140af6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "2c4d718738d1408eabce489f4e140af6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:29.302230 6047 raft_consensus.cc:482] T 3fa99ffab5a44c588738be4476ed0fdc P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:29.302448 6047 raft_consensus.cc:504] T 3fa99ffab5a44c588738be4476ed0fdc P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:29.304188 6047 leader_election.cc:288] T 3fa99ffab5a44c588738be4476ed0fdc P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417), 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937)
I20240501 08:13:29.311151 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fa99ffab5a44c588738be4476ed0fdc" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f" is_pre_election: true
I20240501 08:13:29.311408 5051 raft_consensus.cc:2454] T 3fa99ffab5a44c588738be4476ed0fdc P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 0.
I20240501 08:13:29.311326 5182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fa99ffab5a44c588738be4476ed0fdc" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" is_pre_election: true
I20240501 08:13:29.311722 5182 raft_consensus.cc:2454] T 3fa99ffab5a44c588738be4476ed0fdc P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 0.
I20240501 08:13:29.311903 5247 leader_election.cc:302] T 3fa99ffab5a44c588738be4476ed0fdc P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9808124b81e14d1290ab0a6641a4b836, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:29.312355 6047 raft_consensus.cc:2781] T 3fa99ffab5a44c588738be4476ed0fdc P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:29.312464 6047 raft_consensus.cc:482] T 3fa99ffab5a44c588738be4476ed0fdc P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:29.312552 6047 raft_consensus.cc:3037] T 3fa99ffab5a44c588738be4476ed0fdc P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:29.314829 6047 raft_consensus.cc:504] T 3fa99ffab5a44c588738be4476ed0fdc P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:29.315505 6047 leader_election.cc:288] T 3fa99ffab5a44c588738be4476ed0fdc P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 1 election: Requested vote from peers b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417), 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937)
I20240501 08:13:29.315965 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fa99ffab5a44c588738be4476ed0fdc" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:29.316165 5051 raft_consensus.cc:3037] T 3fa99ffab5a44c588738be4476ed0fdc P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:29.316131 5182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fa99ffab5a44c588738be4476ed0fdc" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:29.316349 5182 raft_consensus.cc:3037] T 3fa99ffab5a44c588738be4476ed0fdc P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:29.318205 5051 raft_consensus.cc:2454] T 3fa99ffab5a44c588738be4476ed0fdc P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 1.
I20240501 08:13:29.318431 5182 raft_consensus.cc:2454] T 3fa99ffab5a44c588738be4476ed0fdc P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 1.
I20240501 08:13:29.318619 5247 leader_election.cc:302] T 3fa99ffab5a44c588738be4476ed0fdc P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9808124b81e14d1290ab0a6641a4b836, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:29.318981 6047 raft_consensus.cc:2781] T 3fa99ffab5a44c588738be4476ed0fdc P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:29.319445 6047 raft_consensus.cc:686] T 3fa99ffab5a44c588738be4476ed0fdc P 9808124b81e14d1290ab0a6641a4b836 [term 1 LEADER]: Becoming Leader. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:29.319988 6047 consensus_queue.cc:237] T 3fa99ffab5a44c588738be4476ed0fdc P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:29.324193 4909 catalog_manager.cc:5484] T 3fa99ffab5a44c588738be4476ed0fdc P 9808124b81e14d1290ab0a6641a4b836 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131). New cstate: current_term: 1 leader_uuid: "9808124b81e14d1290ab0a6641a4b836" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: HEALTHY } } }
W20240501 08:13:29.376488 5488 tablet.cc:2369] T 416ef8d814614a7caba4279e64893b27 P 2c4d718738d1408eabce489f4e140af6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 08:13:29.387909 5358 tablet.cc:2369] T 3fa99ffab5a44c588738be4476ed0fdc P 9808124b81e14d1290ab0a6641a4b836: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 08:13:29.422545 5228 tablet.cc:2369] T 3fa99ffab5a44c588738be4476ed0fdc P 0a952997a9fc4bfb8d7cbc56de6e0598: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 08:13:29.710621 6065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:29.710943 6065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 08:13:29.717149 6047 consensus_queue.cc:1035] T 3fa99ffab5a44c588738be4476ed0fdc P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:13:29.720670 6065 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:13:29.727026 6063 consensus_queue.cc:1035] T 3fa99ffab5a44c588738be4476ed0fdc P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:29.743759 6046 consensus_queue.cc:1035] T 416ef8d814614a7caba4279e64893b27 P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:29.749717 6082 consensus_queue.cc:1035] T 416ef8d814614a7caba4279e64893b27 P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240501 08:13:29.816104 6045 consensus_queue.cc:1035] T 8a96986cf35c4720991739af0626e1e6 P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:29.834444 6045 consensus_queue.cc:1035] T 8a96986cf35c4720991739af0626e1e6 P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 253.943 ms
time per row: 0.126972 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 08:13:31.038569 4902 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33308:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:13:31.047650 5161 tablet_service.cc:1467] Processing CreateTablet for tablet ff3b86014f2940b18707803c38bb5b8f (DEFAULT_TABLE table=rebalance_test_table_1 [id=d37f915a81d843f5896acd04ed1fdf50]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:31.047952 5422 tablet_service.cc:1467] Processing CreateTablet for tablet ff3b86014f2940b18707803c38bb5b8f (DEFAULT_TABLE table=rebalance_test_table_1 [id=d37f915a81d843f5896acd04ed1fdf50]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:31.048161 5161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff3b86014f2940b18707803c38bb5b8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:31.048372 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff3b86014f2940b18707803c38bb5b8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:31.048475 5421 tablet_service.cc:1467] Processing CreateTablet for tablet c1bd10f523444d668736a2ff5e02442c (DEFAULT_TABLE table=rebalance_test_table_1 [id=d37f915a81d843f5896acd04ed1fdf50]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:31.048815 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1bd10f523444d668736a2ff5e02442c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:31.050400 5162 tablet_service.cc:1467] Processing CreateTablet for tablet c1bd10f523444d668736a2ff5e02442c (DEFAULT_TABLE table=rebalance_test_table_1 [id=d37f915a81d843f5896acd04ed1fdf50]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:31.050612 5292 tablet_service.cc:1467] Processing CreateTablet for tablet ff3b86014f2940b18707803c38bb5b8f (DEFAULT_TABLE table=rebalance_test_table_1 [id=d37f915a81d843f5896acd04ed1fdf50]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:31.050807 5162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1bd10f523444d668736a2ff5e02442c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:31.051041 5292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff3b86014f2940b18707803c38bb5b8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:31.051519 5160 tablet_service.cc:1467] Processing CreateTablet for tablet 2b611e855c3941769c4a26bcbe2b9b8c (DEFAULT_TABLE table=rebalance_test_table_1 [id=d37f915a81d843f5896acd04ed1fdf50]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:31.051833 5160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b611e855c3941769c4a26bcbe2b9b8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:31.054339 5291 tablet_service.cc:1467] Processing CreateTablet for tablet c1bd10f523444d668736a2ff5e02442c (DEFAULT_TABLE table=rebalance_test_table_1 [id=d37f915a81d843f5896acd04ed1fdf50]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:31.054747 5291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1bd10f523444d668736a2ff5e02442c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:31.058313 6136 tablet_bootstrap.cc:492] T 2b611e855c3941769c4a26bcbe2b9b8c P 0a952997a9fc4bfb8d7cbc56de6e0598: Bootstrap starting.
I20240501 08:13:31.058382 5290 tablet_service.cc:1467] Processing CreateTablet for tablet 2b611e855c3941769c4a26bcbe2b9b8c (DEFAULT_TABLE table=rebalance_test_table_1 [id=d37f915a81d843f5896acd04ed1fdf50]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:31.058794 5290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b611e855c3941769c4a26bcbe2b9b8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:31.060938 6136 tablet_bootstrap.cc:654] T 2b611e855c3941769c4a26bcbe2b9b8c P 0a952997a9fc4bfb8d7cbc56de6e0598: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:31.062433 5029 tablet_service.cc:1467] Processing CreateTablet for tablet 2b611e855c3941769c4a26bcbe2b9b8c (DEFAULT_TABLE table=rebalance_test_table_1 [id=d37f915a81d843f5896acd04ed1fdf50]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:31.062887 5029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b611e855c3941769c4a26bcbe2b9b8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:31.072901 6136 tablet_bootstrap.cc:492] T 2b611e855c3941769c4a26bcbe2b9b8c P 0a952997a9fc4bfb8d7cbc56de6e0598: No bootstrap required, opened a new log
I20240501 08:13:31.073252 6136 ts_tablet_manager.cc:1367] T 2b611e855c3941769c4a26bcbe2b9b8c P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.010s
I20240501 08:13:31.074409 6136 raft_consensus.cc:348] T 2b611e855c3941769c4a26bcbe2b9b8c P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:31.074710 6136 raft_consensus.cc:374] T 2b611e855c3941769c4a26bcbe2b9b8c P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:31.074862 6136 raft_consensus.cc:729] T 2b611e855c3941769c4a26bcbe2b9b8c P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Initialized, Role: FOLLOWER
I20240501 08:13:31.075208 6136 consensus_queue.cc:260] T 2b611e855c3941769c4a26bcbe2b9b8c P 0a952997a9fc4bfb8d7cbc56de6e0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:31.075949 6136 ts_tablet_manager.cc:1398] T 2b611e855c3941769c4a26bcbe2b9b8c P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:31.076370 6136 tablet_bootstrap.cc:492] T c1bd10f523444d668736a2ff5e02442c P 0a952997a9fc4bfb8d7cbc56de6e0598: Bootstrap starting.
I20240501 08:13:31.078855 6136 tablet_bootstrap.cc:654] T c1bd10f523444d668736a2ff5e02442c P 0a952997a9fc4bfb8d7cbc56de6e0598: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:31.085011 6136 tablet_bootstrap.cc:492] T c1bd10f523444d668736a2ff5e02442c P 0a952997a9fc4bfb8d7cbc56de6e0598: No bootstrap required, opened a new log
I20240501 08:13:31.085292 6136 ts_tablet_manager.cc:1367] T c1bd10f523444d668736a2ff5e02442c P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20240501 08:13:31.086459 6136 raft_consensus.cc:348] T c1bd10f523444d668736a2ff5e02442c P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:31.086886 6136 raft_consensus.cc:374] T c1bd10f523444d668736a2ff5e02442c P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:31.086974 6138 tablet_bootstrap.cc:492] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836: Bootstrap starting.
I20240501 08:13:31.087036 6136 raft_consensus.cc:729] T c1bd10f523444d668736a2ff5e02442c P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Initialized, Role: FOLLOWER
I20240501 08:13:31.087136 6139 tablet_bootstrap.cc:492] T 2b611e855c3941769c4a26bcbe2b9b8c P b6cc443a494e4595b0e7cf1777620a3f: Bootstrap starting.
I20240501 08:13:31.087512 6136 consensus_queue.cc:260] T c1bd10f523444d668736a2ff5e02442c P 0a952997a9fc4bfb8d7cbc56de6e0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:31.088299 6136 ts_tablet_manager.cc:1398] T c1bd10f523444d668736a2ff5e02442c P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 08:13:31.087901 6135 tablet_bootstrap.cc:492] T ff3b86014f2940b18707803c38bb5b8f P 2c4d718738d1408eabce489f4e140af6: Bootstrap starting.
I20240501 08:13:31.088618 6136 tablet_bootstrap.cc:492] T ff3b86014f2940b18707803c38bb5b8f P 0a952997a9fc4bfb8d7cbc56de6e0598: Bootstrap starting.
I20240501 08:13:31.090378 6135 tablet_bootstrap.cc:654] T ff3b86014f2940b18707803c38bb5b8f P 2c4d718738d1408eabce489f4e140af6: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:31.090380 6136 tablet_bootstrap.cc:654] T ff3b86014f2940b18707803c38bb5b8f P 0a952997a9fc4bfb8d7cbc56de6e0598: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:31.092602 6139 tablet_bootstrap.cc:654] T 2b611e855c3941769c4a26bcbe2b9b8c P b6cc443a494e4595b0e7cf1777620a3f: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:31.094094 6138 tablet_bootstrap.cc:654] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:31.094645 6136 tablet_bootstrap.cc:492] T ff3b86014f2940b18707803c38bb5b8f P 0a952997a9fc4bfb8d7cbc56de6e0598: No bootstrap required, opened a new log
I20240501 08:13:31.094857 6136 ts_tablet_manager.cc:1367] T ff3b86014f2940b18707803c38bb5b8f P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.001s
I20240501 08:13:31.095636 6136 raft_consensus.cc:348] T ff3b86014f2940b18707803c38bb5b8f P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:31.095803 6136 raft_consensus.cc:374] T ff3b86014f2940b18707803c38bb5b8f P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:31.095932 6136 raft_consensus.cc:729] T ff3b86014f2940b18707803c38bb5b8f P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Initialized, Role: FOLLOWER
I20240501 08:13:31.096913 6138 tablet_bootstrap.cc:492] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836: No bootstrap required, opened a new log
I20240501 08:13:31.097079 6138 ts_tablet_manager.cc:1367] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20240501 08:13:31.096678 6136 consensus_queue.cc:260] T ff3b86014f2940b18707803c38bb5b8f P 0a952997a9fc4bfb8d7cbc56de6e0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:31.097975 6136 ts_tablet_manager.cc:1398] T ff3b86014f2940b18707803c38bb5b8f P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:13:31.098014 6138 raft_consensus.cc:348] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:31.098634 6135 tablet_bootstrap.cc:492] T ff3b86014f2940b18707803c38bb5b8f P 2c4d718738d1408eabce489f4e140af6: No bootstrap required, opened a new log
I20240501 08:13:31.098719 6138 raft_consensus.cc:374] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:31.098779 6135 ts_tablet_manager.cc:1367] T ff3b86014f2940b18707803c38bb5b8f P 2c4d718738d1408eabce489f4e140af6: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240501 08:13:31.098829 6138 raft_consensus.cc:729] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Initialized, Role: FOLLOWER
I20240501 08:13:31.099064 6138 consensus_queue.cc:260] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:31.099593 6135 raft_consensus.cc:348] T ff3b86014f2940b18707803c38bb5b8f P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:31.099695 6138 ts_tablet_manager.cc:1398] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:31.099792 6135 raft_consensus.cc:374] T ff3b86014f2940b18707803c38bb5b8f P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:31.099891 6135 raft_consensus.cc:729] T ff3b86014f2940b18707803c38bb5b8f P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Initialized, Role: FOLLOWER
I20240501 08:13:31.099990 6138 tablet_bootstrap.cc:492] T 2b611e855c3941769c4a26bcbe2b9b8c P 9808124b81e14d1290ab0a6641a4b836: Bootstrap starting.
I20240501 08:13:31.100167 6135 consensus_queue.cc:260] T ff3b86014f2940b18707803c38bb5b8f P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:31.101619 6135 ts_tablet_manager.cc:1398] T ff3b86014f2940b18707803c38bb5b8f P 2c4d718738d1408eabce489f4e140af6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:13:31.101904 6135 tablet_bootstrap.cc:492] T c1bd10f523444d668736a2ff5e02442c P 2c4d718738d1408eabce489f4e140af6: Bootstrap starting.
I20240501 08:13:31.103883 6135 tablet_bootstrap.cc:654] T c1bd10f523444d668736a2ff5e02442c P 2c4d718738d1408eabce489f4e140af6: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:31.103883 6139 tablet_bootstrap.cc:492] T 2b611e855c3941769c4a26bcbe2b9b8c P b6cc443a494e4595b0e7cf1777620a3f: No bootstrap required, opened a new log
I20240501 08:13:31.104081 6139 ts_tablet_manager.cc:1367] T 2b611e855c3941769c4a26bcbe2b9b8c P b6cc443a494e4595b0e7cf1777620a3f: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240501 08:13:31.105051 6138 tablet_bootstrap.cc:654] T 2b611e855c3941769c4a26bcbe2b9b8c P 9808124b81e14d1290ab0a6641a4b836: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:31.104983 6139 raft_consensus.cc:348] T 2b611e855c3941769c4a26bcbe2b9b8c P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:31.105232 6139 raft_consensus.cc:374] T 2b611e855c3941769c4a26bcbe2b9b8c P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:31.105329 6139 raft_consensus.cc:729] T 2b611e855c3941769c4a26bcbe2b9b8c P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6cc443a494e4595b0e7cf1777620a3f, State: Initialized, Role: FOLLOWER
I20240501 08:13:31.105620 6139 consensus_queue.cc:260] T 2b611e855c3941769c4a26bcbe2b9b8c P b6cc443a494e4595b0e7cf1777620a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:31.106132 6135 tablet_bootstrap.cc:492] T c1bd10f523444d668736a2ff5e02442c P 2c4d718738d1408eabce489f4e140af6: No bootstrap required, opened a new log
I20240501 08:13:31.106317 6135 ts_tablet_manager.cc:1367] T c1bd10f523444d668736a2ff5e02442c P 2c4d718738d1408eabce489f4e140af6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:13:31.107062 6135 raft_consensus.cc:348] T c1bd10f523444d668736a2ff5e02442c P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:31.107241 6139 ts_tablet_manager.cc:1398] T 2b611e855c3941769c4a26bcbe2b9b8c P b6cc443a494e4595b0e7cf1777620a3f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 08:13:31.107291 6135 raft_consensus.cc:374] T c1bd10f523444d668736a2ff5e02442c P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:31.107358 6135 raft_consensus.cc:729] T c1bd10f523444d668736a2ff5e02442c P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Initialized, Role: FOLLOWER
I20240501 08:13:31.107638 6135 consensus_queue.cc:260] T c1bd10f523444d668736a2ff5e02442c P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:31.108178 6135 ts_tablet_manager.cc:1398] T c1bd10f523444d668736a2ff5e02442c P 2c4d718738d1408eabce489f4e140af6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:13:31.108786 6138 tablet_bootstrap.cc:492] T 2b611e855c3941769c4a26bcbe2b9b8c P 9808124b81e14d1290ab0a6641a4b836: No bootstrap required, opened a new log
I20240501 08:13:31.108989 6138 ts_tablet_manager.cc:1367] T 2b611e855c3941769c4a26bcbe2b9b8c P 9808124b81e14d1290ab0a6641a4b836: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20240501 08:13:31.109622 6138 raft_consensus.cc:348] T 2b611e855c3941769c4a26bcbe2b9b8c P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:31.109817 6138 raft_consensus.cc:374] T 2b611e855c3941769c4a26bcbe2b9b8c P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:31.109897 6138 raft_consensus.cc:729] T 2b611e855c3941769c4a26bcbe2b9b8c P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Initialized, Role: FOLLOWER
I20240501 08:13:31.110179 6138 consensus_queue.cc:260] T 2b611e855c3941769c4a26bcbe2b9b8c P 9808124b81e14d1290ab0a6641a4b836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:31.110672 6138 ts_tablet_manager.cc:1398] T 2b611e855c3941769c4a26bcbe2b9b8c P 9808124b81e14d1290ab0a6641a4b836: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:31.111012 6138 tablet_bootstrap.cc:492] T c1bd10f523444d668736a2ff5e02442c P 9808124b81e14d1290ab0a6641a4b836: Bootstrap starting.
I20240501 08:13:31.112684 6138 tablet_bootstrap.cc:654] T c1bd10f523444d668736a2ff5e02442c P 9808124b81e14d1290ab0a6641a4b836: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:31.115252 6138 tablet_bootstrap.cc:492] T c1bd10f523444d668736a2ff5e02442c P 9808124b81e14d1290ab0a6641a4b836: No bootstrap required, opened a new log
I20240501 08:13:31.115469 6138 ts_tablet_manager.cc:1367] T c1bd10f523444d668736a2ff5e02442c P 9808124b81e14d1290ab0a6641a4b836: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240501 08:13:31.116415 6138 raft_consensus.cc:348] T c1bd10f523444d668736a2ff5e02442c P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:31.116660 6138 raft_consensus.cc:374] T c1bd10f523444d668736a2ff5e02442c P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:31.116771 6138 raft_consensus.cc:729] T c1bd10f523444d668736a2ff5e02442c P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Initialized, Role: FOLLOWER
I20240501 08:13:31.117012 6138 consensus_queue.cc:260] T c1bd10f523444d668736a2ff5e02442c P 9808124b81e14d1290ab0a6641a4b836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:31.117704 6138 ts_tablet_manager.cc:1398] T c1bd10f523444d668736a2ff5e02442c P 9808124b81e14d1290ab0a6641a4b836: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:13:31.226603 6106 raft_consensus.cc:482] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:31.226805 6106 raft_consensus.cc:504] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:31.227586 6106 leader_election.cc:288] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937), 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797)
I20240501 08:13:31.228108 5182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff3b86014f2940b18707803c38bb5b8f" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" is_pre_election: true
I20240501 08:13:31.228425 5182 raft_consensus.cc:2454] T ff3b86014f2940b18707803c38bb5b8f P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 0.
I20240501 08:13:31.228914 5246 leader_election.cc:302] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 9808124b81e14d1290ab0a6641a4b836; no voters:
I20240501 08:13:31.229188 6106 raft_consensus.cc:2781] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:31.229303 6106 raft_consensus.cc:482] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:31.229388 6106 raft_consensus.cc:3037] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:31.230054 6129 raft_consensus.cc:482] T 2b611e855c3941769c4a26bcbe2b9b8c P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:31.230180 6129 raft_consensus.cc:504] T 2b611e855c3941769c4a26bcbe2b9b8c P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:31.230892 6129 leader_election.cc:288] T 2b611e855c3941769c4a26bcbe2b9b8c P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983), b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417)
I20240501 08:13:31.232236 6106 raft_consensus.cc:504] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:31.233246 6106 leader_election.cc:288] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 1 election: Requested vote from peers 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937), 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797)
I20240501 08:13:31.233817 5182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff3b86014f2940b18707803c38bb5b8f" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:31.234095 5182 raft_consensus.cc:3037] T ff3b86014f2940b18707803c38bb5b8f P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:31.234603 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b611e855c3941769c4a26bcbe2b9b8c" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f" is_pre_election: true
I20240501 08:13:31.234880 5051 raft_consensus.cc:2454] T 2b611e855c3941769c4a26bcbe2b9b8c P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 0.
I20240501 08:13:31.235423 5117 leader_election.cc:302] T 2b611e855c3941769c4a26bcbe2b9b8c P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:31.235723 6129 raft_consensus.cc:2781] T 2b611e855c3941769c4a26bcbe2b9b8c P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:31.235857 6129 raft_consensus.cc:482] T 2b611e855c3941769c4a26bcbe2b9b8c P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:31.235947 6129 raft_consensus.cc:3037] T 2b611e855c3941769c4a26bcbe2b9b8c P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:31.237066 5182 raft_consensus.cc:2454] T ff3b86014f2940b18707803c38bb5b8f P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 1.
I20240501 08:13:31.237522 5246 leader_election.cc:302] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 9808124b81e14d1290ab0a6641a4b836; no voters:
I20240501 08:13:31.237962 6106 raft_consensus.cc:2781] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:31.238770 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff3b86014f2940b18707803c38bb5b8f" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4d718738d1408eabce489f4e140af6" is_pre_election: true
I20240501 08:13:31.238826 6129 raft_consensus.cc:504] T 2b611e855c3941769c4a26bcbe2b9b8c P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:31.239081 5442 raft_consensus.cc:2454] T ff3b86014f2940b18707803c38bb5b8f P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 0.
I20240501 08:13:31.239514 6129 leader_election.cc:288] T 2b611e855c3941769c4a26bcbe2b9b8c P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 election: Requested vote from peers 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983), b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417)
I20240501 08:13:31.239550 5441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff3b86014f2940b18707803c38bb5b8f" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:31.239750 5441 raft_consensus.cc:3037] T ff3b86014f2940b18707803c38bb5b8f P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:31.239842 6106 raft_consensus.cc:686] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836 [term 1 LEADER]: Becoming Leader. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:31.240190 6106 consensus_queue.cc:237] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:31.240748 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b611e855c3941769c4a26bcbe2b9b8c" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:31.240955 5051 raft_consensus.cc:3037] T 2b611e855c3941769c4a26bcbe2b9b8c P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:31.243495 5051 raft_consensus.cc:2454] T 2b611e855c3941769c4a26bcbe2b9b8c P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 1.
I20240501 08:13:31.243450 5441 raft_consensus.cc:2454] T ff3b86014f2940b18707803c38bb5b8f P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 1.
I20240501 08:13:31.244079 6130 raft_consensus.cc:482] T c1bd10f523444d668736a2ff5e02442c P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:31.244220 6130 raft_consensus.cc:504] T c1bd10f523444d668736a2ff5e02442c P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:31.244537 5117 leader_election.cc:302] T 2b611e855c3941769c4a26bcbe2b9b8c P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:31.244843 6129 raft_consensus.cc:2781] T 2b611e855c3941769c4a26bcbe2b9b8c P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:31.245117 6130 leader_election.cc:288] T c1bd10f523444d668736a2ff5e02442c P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983), 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937)
I20240501 08:13:31.245601 5312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1bd10f523444d668736a2ff5e02442c" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808124b81e14d1290ab0a6641a4b836" is_pre_election: true
I20240501 08:13:31.245894 5312 raft_consensus.cc:2454] T c1bd10f523444d668736a2ff5e02442c P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 0.
I20240501 08:13:31.246392 5377 leader_election.cc:302] T c1bd10f523444d668736a2ff5e02442c P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4d718738d1408eabce489f4e140af6, 9808124b81e14d1290ab0a6641a4b836; no voters:
I20240501 08:13:31.246857 6130 raft_consensus.cc:2781] T c1bd10f523444d668736a2ff5e02442c P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:31.246965 6130 raft_consensus.cc:482] T c1bd10f523444d668736a2ff5e02442c P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:31.247063 6130 raft_consensus.cc:3037] T c1bd10f523444d668736a2ff5e02442c P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:31.247220 5312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b611e855c3941769c4a26bcbe2b9b8c" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808124b81e14d1290ab0a6641a4b836" is_pre_election: true
I20240501 08:13:31.247112 4902 catalog_manager.cc:5484] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131). New cstate: current_term: 1 leader_uuid: "9808124b81e14d1290ab0a6641a4b836" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:31.247416 5312 raft_consensus.cc:2454] T 2b611e855c3941769c4a26bcbe2b9b8c P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 0.
I20240501 08:13:31.248124 5312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b611e855c3941769c4a26bcbe2b9b8c" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
I20240501 08:13:31.248312 5312 raft_consensus.cc:3037] T 2b611e855c3941769c4a26bcbe2b9b8c P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:31.249997 6130 raft_consensus.cc:504] T c1bd10f523444d668736a2ff5e02442c P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:31.250360 5312 raft_consensus.cc:2454] T 2b611e855c3941769c4a26bcbe2b9b8c P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 1.
I20240501 08:13:31.251386 5312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1bd10f523444d668736a2ff5e02442c" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
I20240501 08:13:31.251569 5312 raft_consensus.cc:3037] T c1bd10f523444d668736a2ff5e02442c P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:31.251448 6130 leader_election.cc:288] T c1bd10f523444d668736a2ff5e02442c P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 1 election: Requested vote from peers 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983), 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937)
I20240501 08:13:31.253969 5312 raft_consensus.cc:2454] T c1bd10f523444d668736a2ff5e02442c P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 1.
I20240501 08:13:31.254200 6129 raft_consensus.cc:686] T 2b611e855c3941769c4a26bcbe2b9b8c P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Becoming Leader. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Running, Role: LEADER
I20240501 08:13:31.254422 5377 leader_election.cc:302] T c1bd10f523444d668736a2ff5e02442c P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4d718738d1408eabce489f4e140af6, 9808124b81e14d1290ab0a6641a4b836; no voters:
I20240501 08:13:31.254734 6130 raft_consensus.cc:2781] T c1bd10f523444d668736a2ff5e02442c P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:31.255029 5182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1bd10f523444d668736a2ff5e02442c" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" is_pre_election: true
I20240501 08:13:31.255163 5181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1bd10f523444d668736a2ff5e02442c" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:31.255260 6130 raft_consensus.cc:686] T c1bd10f523444d668736a2ff5e02442c P 2c4d718738d1408eabce489f4e140af6 [term 1 LEADER]: Becoming Leader. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:31.255262 5182 raft_consensus.cc:2454] T c1bd10f523444d668736a2ff5e02442c P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 0.
I20240501 08:13:31.254822 6129 consensus_queue.cc:237] T 2b611e855c3941769c4a26bcbe2b9b8c P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:31.255602 6130 consensus_queue.cc:237] T c1bd10f523444d668736a2ff5e02442c P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:31.258481 4909 catalog_manager.cc:5484] T 2b611e855c3941769c4a26bcbe2b9b8c P 0a952997a9fc4bfb8d7cbc56de6e0598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130). New cstate: current_term: 1 leader_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:31.258486 4902 catalog_manager.cc:5484] T c1bd10f523444d668736a2ff5e02442c P 2c4d718738d1408eabce489f4e140af6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c4d718738d1408eabce489f4e140af6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "2c4d718738d1408eabce489f4e140af6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: HEALTHY } } }
W20240501 08:13:31.618943 6162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:31.619324 6162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:13:31.626956 6162 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:13:31.655406 6106 consensus_queue.cc:1035] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240501 08:13:31.661875 5182 raft_consensus.cc:3037] T c1bd10f523444d668736a2ff5e02442c P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:31.667826 6130 consensus_queue.cc:1035] T c1bd10f523444d668736a2ff5e02442c P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:31.670089 6155 consensus_queue.cc:1035] T 2b611e855c3941769c4a26bcbe2b9b8c P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:31.676810 6106 consensus_queue.cc:1035] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:31.711055 6130 consensus_queue.cc:1035] T c1bd10f523444d668736a2ff5e02442c P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:31.733493 6155 consensus_queue.cc:1035] T 2b611e855c3941769c4a26bcbe2b9b8c P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 256.571 ms
time per row: 0.128285 ms
I20240501 08:13:32.113868 4902 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33308:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:13:32.123070 5161 tablet_service.cc:1467] Processing CreateTablet for tablet e9e30e05c0ef498288f35706978c981b (DEFAULT_TABLE table=rebalance_test_table_2 [id=89aa2fbdbeed4996ac740ceeb0e53fce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:32.123185 5029 tablet_service.cc:1467] Processing CreateTablet for tablet e9e30e05c0ef498288f35706978c981b (DEFAULT_TABLE table=rebalance_test_table_2 [id=89aa2fbdbeed4996ac740ceeb0e53fce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:32.123562 5161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9e30e05c0ef498288f35706978c981b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:32.123638 5029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9e30e05c0ef498288f35706978c981b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:32.123642 5292 tablet_service.cc:1467] Processing CreateTablet for tablet 14d9bd97e4674e0ebdd90bf45b5679cb (DEFAULT_TABLE table=rebalance_test_table_2 [id=89aa2fbdbeed4996ac740ceeb0e53fce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:32.124009 5292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14d9bd97e4674e0ebdd90bf45b5679cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:32.127285 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 14d9bd97e4674e0ebdd90bf45b5679cb (DEFAULT_TABLE table=rebalance_test_table_2 [id=89aa2fbdbeed4996ac740ceeb0e53fce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:32.127786 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14d9bd97e4674e0ebdd90bf45b5679cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:32.128090 5162 tablet_service.cc:1467] Processing CreateTablet for tablet d381c4733f234e59b8eb6cf44166e007 (DEFAULT_TABLE table=rebalance_test_table_2 [id=89aa2fbdbeed4996ac740ceeb0e53fce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:32.128599 5162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d381c4733f234e59b8eb6cf44166e007. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:32.130575 5420 tablet_service.cc:1467] Processing CreateTablet for tablet d381c4733f234e59b8eb6cf44166e007 (DEFAULT_TABLE table=rebalance_test_table_2 [id=89aa2fbdbeed4996ac740ceeb0e53fce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:32.131042 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d381c4733f234e59b8eb6cf44166e007. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:32.134101 5422 tablet_service.cc:1467] Processing CreateTablet for tablet e9e30e05c0ef498288f35706978c981b (DEFAULT_TABLE table=rebalance_test_table_2 [id=89aa2fbdbeed4996ac740ceeb0e53fce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:32.134533 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9e30e05c0ef498288f35706978c981b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:32.135366 5030 tablet_service.cc:1467] Processing CreateTablet for tablet 14d9bd97e4674e0ebdd90bf45b5679cb (DEFAULT_TABLE table=rebalance_test_table_2 [id=89aa2fbdbeed4996ac740ceeb0e53fce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:32.135787 5030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14d9bd97e4674e0ebdd90bf45b5679cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:32.138381 5031 tablet_service.cc:1467] Processing CreateTablet for tablet d381c4733f234e59b8eb6cf44166e007 (DEFAULT_TABLE table=rebalance_test_table_2 [id=89aa2fbdbeed4996ac740ceeb0e53fce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:32.138787 5031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d381c4733f234e59b8eb6cf44166e007. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:32.139137 6227 tablet_bootstrap.cc:492] T 14d9bd97e4674e0ebdd90bf45b5679cb P 9808124b81e14d1290ab0a6641a4b836: Bootstrap starting.
I20240501 08:13:32.141845 6227 tablet_bootstrap.cc:654] T 14d9bd97e4674e0ebdd90bf45b5679cb P 9808124b81e14d1290ab0a6641a4b836: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:32.160583 6230 tablet_bootstrap.cc:492] T e9e30e05c0ef498288f35706978c981b P b6cc443a494e4595b0e7cf1777620a3f: Bootstrap starting.
I20240501 08:13:32.160583 6229 tablet_bootstrap.cc:492] T e9e30e05c0ef498288f35706978c981b P 2c4d718738d1408eabce489f4e140af6: Bootstrap starting.
I20240501 08:13:32.162586 6229 tablet_bootstrap.cc:654] T e9e30e05c0ef498288f35706978c981b P 2c4d718738d1408eabce489f4e140af6: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:32.162590 6230 tablet_bootstrap.cc:654] T e9e30e05c0ef498288f35706978c981b P b6cc443a494e4595b0e7cf1777620a3f: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:32.165449 6226 tablet_bootstrap.cc:492] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598: Bootstrap starting.
I20240501 08:13:32.165550 6227 tablet_bootstrap.cc:492] T 14d9bd97e4674e0ebdd90bf45b5679cb P 9808124b81e14d1290ab0a6641a4b836: No bootstrap required, opened a new log
I20240501 08:13:32.165710 6227 ts_tablet_manager.cc:1367] T 14d9bd97e4674e0ebdd90bf45b5679cb P 9808124b81e14d1290ab0a6641a4b836: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.015s
I20240501 08:13:32.166469 6227 raft_consensus.cc:348] T 14d9bd97e4674e0ebdd90bf45b5679cb P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:32.166867 6227 raft_consensus.cc:374] T 14d9bd97e4674e0ebdd90bf45b5679cb P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:32.166963 6227 raft_consensus.cc:729] T 14d9bd97e4674e0ebdd90bf45b5679cb P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Initialized, Role: FOLLOWER
I20240501 08:13:32.167229 6227 consensus_queue.cc:260] T 14d9bd97e4674e0ebdd90bf45b5679cb P 9808124b81e14d1290ab0a6641a4b836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:32.168076 6226 tablet_bootstrap.cc:654] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:32.168445 6227 ts_tablet_manager.cc:1398] T 14d9bd97e4674e0ebdd90bf45b5679cb P 9808124b81e14d1290ab0a6641a4b836: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240501 08:13:32.169627 6229 tablet_bootstrap.cc:492] T e9e30e05c0ef498288f35706978c981b P 2c4d718738d1408eabce489f4e140af6: No bootstrap required, opened a new log
I20240501 08:13:32.169852 6229 ts_tablet_manager.cc:1367] T e9e30e05c0ef498288f35706978c981b P 2c4d718738d1408eabce489f4e140af6: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240501 08:13:32.170241 6230 tablet_bootstrap.cc:492] T e9e30e05c0ef498288f35706978c981b P b6cc443a494e4595b0e7cf1777620a3f: No bootstrap required, opened a new log
I20240501 08:13:32.170401 6230 ts_tablet_manager.cc:1367] T e9e30e05c0ef498288f35706978c981b P b6cc443a494e4595b0e7cf1777620a3f: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240501 08:13:32.170957 6229 raft_consensus.cc:348] T e9e30e05c0ef498288f35706978c981b P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:32.171257 6229 raft_consensus.cc:374] T e9e30e05c0ef498288f35706978c981b P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:32.171187 6230 raft_consensus.cc:348] T e9e30e05c0ef498288f35706978c981b P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:32.171386 6229 raft_consensus.cc:729] T e9e30e05c0ef498288f35706978c981b P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Initialized, Role: FOLLOWER
I20240501 08:13:32.171438 6230 raft_consensus.cc:374] T e9e30e05c0ef498288f35706978c981b P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:32.171523 6230 raft_consensus.cc:729] T e9e30e05c0ef498288f35706978c981b P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6cc443a494e4595b0e7cf1777620a3f, State: Initialized, Role: FOLLOWER
I20240501 08:13:32.171751 6230 consensus_queue.cc:260] T e9e30e05c0ef498288f35706978c981b P b6cc443a494e4595b0e7cf1777620a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:32.171789 6229 consensus_queue.cc:260] T e9e30e05c0ef498288f35706978c981b P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:32.172358 6230 ts_tablet_manager.cc:1398] T e9e30e05c0ef498288f35706978c981b P b6cc443a494e4595b0e7cf1777620a3f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:13:32.172667 6230 tablet_bootstrap.cc:492] T 14d9bd97e4674e0ebdd90bf45b5679cb P b6cc443a494e4595b0e7cf1777620a3f: Bootstrap starting.
I20240501 08:13:32.174149 6229 ts_tablet_manager.cc:1398] T e9e30e05c0ef498288f35706978c981b P 2c4d718738d1408eabce489f4e140af6: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:13:32.174499 6229 tablet_bootstrap.cc:492] T 14d9bd97e4674e0ebdd90bf45b5679cb P 2c4d718738d1408eabce489f4e140af6: Bootstrap starting.
I20240501 08:13:32.174706 6230 tablet_bootstrap.cc:654] T 14d9bd97e4674e0ebdd90bf45b5679cb P b6cc443a494e4595b0e7cf1777620a3f: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:32.176393 6229 tablet_bootstrap.cc:654] T 14d9bd97e4674e0ebdd90bf45b5679cb P 2c4d718738d1408eabce489f4e140af6: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:32.179106 6226 tablet_bootstrap.cc:492] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598: No bootstrap required, opened a new log
I20240501 08:13:32.179315 6226 ts_tablet_manager.cc:1367] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20240501 08:13:32.180115 6230 tablet_bootstrap.cc:492] T 14d9bd97e4674e0ebdd90bf45b5679cb P b6cc443a494e4595b0e7cf1777620a3f: No bootstrap required, opened a new log
I20240501 08:13:32.180238 6229 tablet_bootstrap.cc:492] T 14d9bd97e4674e0ebdd90bf45b5679cb P 2c4d718738d1408eabce489f4e140af6: No bootstrap required, opened a new log
I20240501 08:13:32.180344 6230 ts_tablet_manager.cc:1367] T 14d9bd97e4674e0ebdd90bf45b5679cb P b6cc443a494e4595b0e7cf1777620a3f: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240501 08:13:32.180404 6229 ts_tablet_manager.cc:1367] T 14d9bd97e4674e0ebdd90bf45b5679cb P 2c4d718738d1408eabce489f4e140af6: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240501 08:13:32.180289 6226 raft_consensus.cc:348] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:32.180503 6226 raft_consensus.cc:374] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:32.180615 6226 raft_consensus.cc:729] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Initialized, Role: FOLLOWER
I20240501 08:13:32.180861 6226 consensus_queue.cc:260] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:32.181128 6229 raft_consensus.cc:348] T 14d9bd97e4674e0ebdd90bf45b5679cb P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:32.181162 6230 raft_consensus.cc:348] T 14d9bd97e4674e0ebdd90bf45b5679cb P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:32.181330 6229 raft_consensus.cc:374] T 14d9bd97e4674e0ebdd90bf45b5679cb P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:32.181401 6229 raft_consensus.cc:729] T 14d9bd97e4674e0ebdd90bf45b5679cb P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Initialized, Role: FOLLOWER
I20240501 08:13:32.181403 6230 raft_consensus.cc:374] T 14d9bd97e4674e0ebdd90bf45b5679cb P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:32.181496 6230 raft_consensus.cc:729] T 14d9bd97e4674e0ebdd90bf45b5679cb P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6cc443a494e4595b0e7cf1777620a3f, State: Initialized, Role: FOLLOWER
I20240501 08:13:32.181631 6226 ts_tablet_manager.cc:1398] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:32.181682 6229 consensus_queue.cc:260] T 14d9bd97e4674e0ebdd90bf45b5679cb P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:32.181763 6230 consensus_queue.cc:260] T 14d9bd97e4674e0ebdd90bf45b5679cb P b6cc443a494e4595b0e7cf1777620a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:32.182044 6226 tablet_bootstrap.cc:492] T d381c4733f234e59b8eb6cf44166e007 P 0a952997a9fc4bfb8d7cbc56de6e0598: Bootstrap starting.
I20240501 08:13:32.182293 6230 ts_tablet_manager.cc:1398] T 14d9bd97e4674e0ebdd90bf45b5679cb P b6cc443a494e4595b0e7cf1777620a3f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:32.182790 6229 ts_tablet_manager.cc:1398] T 14d9bd97e4674e0ebdd90bf45b5679cb P 2c4d718738d1408eabce489f4e140af6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:13:32.183090 6229 tablet_bootstrap.cc:492] T d381c4733f234e59b8eb6cf44166e007 P 2c4d718738d1408eabce489f4e140af6: Bootstrap starting.
I20240501 08:13:32.185202 6230 tablet_bootstrap.cc:492] T d381c4733f234e59b8eb6cf44166e007 P b6cc443a494e4595b0e7cf1777620a3f: Bootstrap starting.
I20240501 08:13:32.185271 6229 tablet_bootstrap.cc:654] T d381c4733f234e59b8eb6cf44166e007 P 2c4d718738d1408eabce489f4e140af6: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:32.187156 6230 tablet_bootstrap.cc:654] T d381c4733f234e59b8eb6cf44166e007 P b6cc443a494e4595b0e7cf1777620a3f: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:32.190840 6226 tablet_bootstrap.cc:654] T d381c4733f234e59b8eb6cf44166e007 P 0a952997a9fc4bfb8d7cbc56de6e0598: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:32.191289 6230 tablet_bootstrap.cc:492] T d381c4733f234e59b8eb6cf44166e007 P b6cc443a494e4595b0e7cf1777620a3f: No bootstrap required, opened a new log
I20240501 08:13:32.191457 6230 ts_tablet_manager.cc:1367] T d381c4733f234e59b8eb6cf44166e007 P b6cc443a494e4595b0e7cf1777620a3f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20240501 08:13:32.191486 6229 tablet_bootstrap.cc:492] T d381c4733f234e59b8eb6cf44166e007 P 2c4d718738d1408eabce489f4e140af6: No bootstrap required, opened a new log
I20240501 08:13:32.191622 6229 ts_tablet_manager.cc:1367] T d381c4733f234e59b8eb6cf44166e007 P 2c4d718738d1408eabce489f4e140af6: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20240501 08:13:32.192292 6230 raft_consensus.cc:348] T d381c4733f234e59b8eb6cf44166e007 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:32.192356 6229 raft_consensus.cc:348] T d381c4733f234e59b8eb6cf44166e007 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:32.192536 6230 raft_consensus.cc:374] T d381c4733f234e59b8eb6cf44166e007 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:32.192543 6229 raft_consensus.cc:374] T d381c4733f234e59b8eb6cf44166e007 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:32.192593 6226 tablet_bootstrap.cc:492] T d381c4733f234e59b8eb6cf44166e007 P 0a952997a9fc4bfb8d7cbc56de6e0598: No bootstrap required, opened a new log
I20240501 08:13:32.192634 6229 raft_consensus.cc:729] T d381c4733f234e59b8eb6cf44166e007 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Initialized, Role: FOLLOWER
I20240501 08:13:32.192631 6230 raft_consensus.cc:729] T d381c4733f234e59b8eb6cf44166e007 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6cc443a494e4595b0e7cf1777620a3f, State: Initialized, Role: FOLLOWER
I20240501 08:13:32.192754 6226 ts_tablet_manager.cc:1367] T d381c4733f234e59b8eb6cf44166e007 P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240501 08:13:32.192893 6229 consensus_queue.cc:260] T d381c4733f234e59b8eb6cf44166e007 P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:32.192893 6230 consensus_queue.cc:260] T d381c4733f234e59b8eb6cf44166e007 P b6cc443a494e4595b0e7cf1777620a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:32.193475 6229 ts_tablet_manager.cc:1398] T d381c4733f234e59b8eb6cf44166e007 P 2c4d718738d1408eabce489f4e140af6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:13:32.193575 6226 raft_consensus.cc:348] T d381c4733f234e59b8eb6cf44166e007 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:32.193795 6226 raft_consensus.cc:374] T d381c4733f234e59b8eb6cf44166e007 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:32.193877 6226 raft_consensus.cc:729] T d381c4733f234e59b8eb6cf44166e007 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Initialized, Role: FOLLOWER
I20240501 08:13:32.194134 6226 consensus_queue.cc:260] T d381c4733f234e59b8eb6cf44166e007 P 0a952997a9fc4bfb8d7cbc56de6e0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:32.194959 6230 ts_tablet_manager.cc:1398] T d381c4733f234e59b8eb6cf44166e007 P b6cc443a494e4595b0e7cf1777620a3f: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 08:13:32.195576 6226 ts_tablet_manager.cc:1398] T d381c4733f234e59b8eb6cf44166e007 P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:13:32.198570 6130 raft_consensus.cc:482] T d381c4733f234e59b8eb6cf44166e007 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:32.198763 6130 raft_consensus.cc:504] T d381c4733f234e59b8eb6cf44166e007 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:32.199357 6130 leader_election.cc:288] T d381c4733f234e59b8eb6cf44166e007 P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417), 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937)
I20240501 08:13:32.199831 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d381c4733f234e59b8eb6cf44166e007" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f" is_pre_election: true
I20240501 08:13:32.199883 5181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d381c4733f234e59b8eb6cf44166e007" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" is_pre_election: true
I20240501 08:13:32.200083 5051 raft_consensus.cc:2454] T d381c4733f234e59b8eb6cf44166e007 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 0.
I20240501 08:13:32.200098 5181 raft_consensus.cc:2454] T d381c4733f234e59b8eb6cf44166e007 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 0.
I20240501 08:13:32.200497 5376 leader_election.cc:302] T d381c4733f234e59b8eb6cf44166e007 P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 2c4d718738d1408eabce489f4e140af6, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:32.200829 6130 raft_consensus.cc:2781] T d381c4733f234e59b8eb6cf44166e007 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:32.200929 6130 raft_consensus.cc:482] T d381c4733f234e59b8eb6cf44166e007 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:32.201042 6130 raft_consensus.cc:3037] T d381c4733f234e59b8eb6cf44166e007 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:32.203153 6130 raft_consensus.cc:504] T d381c4733f234e59b8eb6cf44166e007 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:32.203675 6130 leader_election.cc:288] T d381c4733f234e59b8eb6cf44166e007 P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 1 election: Requested vote from peers b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417), 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937)
I20240501 08:13:32.204130 5181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d381c4733f234e59b8eb6cf44166e007" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:32.204175 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d381c4733f234e59b8eb6cf44166e007" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:32.204361 5181 raft_consensus.cc:3037] T d381c4733f234e59b8eb6cf44166e007 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:32.204386 5051 raft_consensus.cc:3037] T d381c4733f234e59b8eb6cf44166e007 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:32.207197 5181 raft_consensus.cc:2454] T d381c4733f234e59b8eb6cf44166e007 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 1.
I20240501 08:13:32.207237 5051 raft_consensus.cc:2454] T d381c4733f234e59b8eb6cf44166e007 P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 1.
I20240501 08:13:32.207628 5376 leader_election.cc:302] T d381c4733f234e59b8eb6cf44166e007 P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 2c4d718738d1408eabce489f4e140af6; no voters:
I20240501 08:13:32.207970 6130 raft_consensus.cc:2781] T d381c4733f234e59b8eb6cf44166e007 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:32.208137 6130 raft_consensus.cc:686] T d381c4733f234e59b8eb6cf44166e007 P 2c4d718738d1408eabce489f4e140af6 [term 1 LEADER]: Becoming Leader. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:32.208413 6130 consensus_queue.cc:237] T d381c4733f234e59b8eb6cf44166e007 P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:32.211251 4902 catalog_manager.cc:5484] T d381c4733f234e59b8eb6cf44166e007 P 2c4d718738d1408eabce489f4e140af6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c4d718738d1408eabce489f4e140af6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "2c4d718738d1408eabce489f4e140af6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:32.259092 6179 raft_consensus.cc:482] T 14d9bd97e4674e0ebdd90bf45b5679cb P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:32.259317 6179 raft_consensus.cc:504] T 14d9bd97e4674e0ebdd90bf45b5679cb P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:32.261005 6179 leader_election.cc:288] T 14d9bd97e4674e0ebdd90bf45b5679cb P b6cc443a494e4595b0e7cf1777620a3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983), 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797)
I20240501 08:13:32.267675 5441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14d9bd97e4674e0ebdd90bf45b5679cb" candidate_uuid: "b6cc443a494e4595b0e7cf1777620a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4d718738d1408eabce489f4e140af6" is_pre_election: true
I20240501 08:13:32.267695 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14d9bd97e4674e0ebdd90bf45b5679cb" candidate_uuid: "b6cc443a494e4595b0e7cf1777620a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808124b81e14d1290ab0a6641a4b836" is_pre_election: true
I20240501 08:13:32.267920 5311 raft_consensus.cc:2454] T 14d9bd97e4674e0ebdd90bf45b5679cb P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6cc443a494e4595b0e7cf1777620a3f in term 0.
I20240501 08:13:32.267935 5441 raft_consensus.cc:2454] T 14d9bd97e4674e0ebdd90bf45b5679cb P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6cc443a494e4595b0e7cf1777620a3f in term 0.
I20240501 08:13:32.268399 4987 leader_election.cc:302] T 14d9bd97e4674e0ebdd90bf45b5679cb P b6cc443a494e4595b0e7cf1777620a3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4d718738d1408eabce489f4e140af6, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:32.268757 6179 raft_consensus.cc:2781] T 14d9bd97e4674e0ebdd90bf45b5679cb P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:32.268864 6179 raft_consensus.cc:482] T 14d9bd97e4674e0ebdd90bf45b5679cb P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:32.268977 6179 raft_consensus.cc:3037] T 14d9bd97e4674e0ebdd90bf45b5679cb P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:32.271184 6179 raft_consensus.cc:504] T 14d9bd97e4674e0ebdd90bf45b5679cb P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:32.271812 6179 leader_election.cc:288] T 14d9bd97e4674e0ebdd90bf45b5679cb P b6cc443a494e4595b0e7cf1777620a3f [CANDIDATE]: Term 1 election: Requested vote from peers 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983), 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797)
I20240501 08:13:32.272243 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14d9bd97e4674e0ebdd90bf45b5679cb" candidate_uuid: "b6cc443a494e4595b0e7cf1777620a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
I20240501 08:13:32.272379 5441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14d9bd97e4674e0ebdd90bf45b5679cb" candidate_uuid: "b6cc443a494e4595b0e7cf1777620a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:32.272459 5311 raft_consensus.cc:3037] T 14d9bd97e4674e0ebdd90bf45b5679cb P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:32.272598 5441 raft_consensus.cc:3037] T 14d9bd97e4674e0ebdd90bf45b5679cb P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:32.275403 5441 raft_consensus.cc:2454] T 14d9bd97e4674e0ebdd90bf45b5679cb P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6cc443a494e4595b0e7cf1777620a3f in term 1.
I20240501 08:13:32.275416 5311 raft_consensus.cc:2454] T 14d9bd97e4674e0ebdd90bf45b5679cb P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6cc443a494e4595b0e7cf1777620a3f in term 1.
I20240501 08:13:32.275871 4986 leader_election.cc:302] T 14d9bd97e4674e0ebdd90bf45b5679cb P b6cc443a494e4595b0e7cf1777620a3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9808124b81e14d1290ab0a6641a4b836, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:32.276185 6179 raft_consensus.cc:2781] T 14d9bd97e4674e0ebdd90bf45b5679cb P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:32.276567 6179 raft_consensus.cc:686] T 14d9bd97e4674e0ebdd90bf45b5679cb P b6cc443a494e4595b0e7cf1777620a3f [term 1 LEADER]: Becoming Leader. State: Replica: b6cc443a494e4595b0e7cf1777620a3f, State: Running, Role: LEADER
I20240501 08:13:32.276979 6179 consensus_queue.cc:237] T 14d9bd97e4674e0ebdd90bf45b5679cb P b6cc443a494e4595b0e7cf1777620a3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:32.280468 4902 catalog_manager.cc:5484] T 14d9bd97e4674e0ebdd90bf45b5679cb P b6cc443a494e4595b0e7cf1777620a3f reported cstate change: term changed from 0 to 1, leader changed from <none> to b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129). New cstate: current_term: 1 leader_uuid: "b6cc443a494e4595b0e7cf1777620a3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:32.367946 6129 raft_consensus.cc:482] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:32.368197 6129 raft_consensus.cc:504] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:32.368801 6129 leader_election.cc:288] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417), 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797)
I20240501 08:13:32.369345 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e30e05c0ef498288f35706978c981b" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f" is_pre_election: true
I20240501 08:13:32.369383 5441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e30e05c0ef498288f35706978c981b" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4d718738d1408eabce489f4e140af6" is_pre_election: true
I20240501 08:13:32.369642 5441 raft_consensus.cc:2454] T e9e30e05c0ef498288f35706978c981b P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 0.
I20240501 08:13:32.369664 5051 raft_consensus.cc:2454] T e9e30e05c0ef498288f35706978c981b P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 0.
I20240501 08:13:32.370092 5118 leader_election.cc:302] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 2c4d718738d1408eabce489f4e140af6; no voters:
I20240501 08:13:32.370376 6129 raft_consensus.cc:2781] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:32.370460 6129 raft_consensus.cc:482] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:32.370563 6129 raft_consensus.cc:3037] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:32.372473 6129 raft_consensus.cc:504] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:32.373147 6129 leader_election.cc:288] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 election: Requested vote from peers b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417), 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797)
I20240501 08:13:32.373598 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e30e05c0ef498288f35706978c981b" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:32.373656 5441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e30e05c0ef498288f35706978c981b" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:32.373836 5441 raft_consensus.cc:3037] T e9e30e05c0ef498288f35706978c981b P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:32.373840 5051 raft_consensus.cc:3037] T e9e30e05c0ef498288f35706978c981b P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:32.376644 5441 raft_consensus.cc:2454] T e9e30e05c0ef498288f35706978c981b P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 1.
I20240501 08:13:32.376631 5051 raft_consensus.cc:2454] T e9e30e05c0ef498288f35706978c981b P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 1.
I20240501 08:13:32.377117 5118 leader_election.cc:302] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 2c4d718738d1408eabce489f4e140af6; no voters:
I20240501 08:13:32.377403 6129 raft_consensus.cc:2781] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:32.377573 6129 raft_consensus.cc:686] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Becoming Leader. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Running, Role: LEADER
I20240501 08:13:32.377831 6129 consensus_queue.cc:237] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:32.380992 4902 catalog_manager.cc:5484] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130). New cstate: current_term: 1 leader_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:32.595217 6130 consensus_queue.cc:1035] T d381c4733f234e59b8eb6cf44166e007 P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:32.600158 6130 consensus_queue.cc:1035] T d381c4733f234e59b8eb6cf44166e007 P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:13:32.748528 6245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:32.748888 6245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:13:32.756525 6245 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:13:32.783414 6179 consensus_queue.cc:1035] T 14d9bd97e4674e0ebdd90bf45b5679cb P b6cc443a494e4595b0e7cf1777620a3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:32.791172 6262 consensus_queue.cc:1035] T 14d9bd97e4674e0ebdd90bf45b5679cb P b6cc443a494e4595b0e7cf1777620a3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240501 08:13:32.894590 6129 consensus_queue.cc:1035] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:32.950189 6289 consensus_queue.cc:1035] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 321.367 ms
time per row: 0.160684 ms
I20240501 08:13:33.230952 4905 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33308:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:13:33.239933 5292 tablet_service.cc:1467] Processing CreateTablet for tablet 51b04d461eed462fa3660c27d75d05b7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=23044fbda733476b84ec4cacff9be572]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:33.240450 5292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51b04d461eed462fa3660c27d75d05b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:33.240535 5030 tablet_service.cc:1467] Processing CreateTablet for tablet a3445b99b36444bb811486be77100479 (DEFAULT_TABLE table=rebalance_test_table_3 [id=23044fbda733476b84ec4cacff9be572]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:33.240772 5422 tablet_service.cc:1467] Processing CreateTablet for tablet a3445b99b36444bb811486be77100479 (DEFAULT_TABLE table=rebalance_test_table_3 [id=23044fbda733476b84ec4cacff9be572]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:33.241061 5030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3445b99b36444bb811486be77100479. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:33.240553 5029 tablet_service.cc:1467] Processing CreateTablet for tablet 51b04d461eed462fa3660c27d75d05b7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=23044fbda733476b84ec4cacff9be572]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:33.241197 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3445b99b36444bb811486be77100479. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:33.241474 5029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51b04d461eed462fa3660c27d75d05b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:33.247005 5420 tablet_service.cc:1467] Processing CreateTablet for tablet 4c29ea3b807d4da1b970246c64797a91 (DEFAULT_TABLE table=rebalance_test_table_3 [id=23044fbda733476b84ec4cacff9be572]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:33.247071 5161 tablet_service.cc:1467] Processing CreateTablet for tablet 51b04d461eed462fa3660c27d75d05b7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=23044fbda733476b84ec4cacff9be572]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:33.247504 5161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51b04d461eed462fa3660c27d75d05b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:33.247572 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c29ea3b807d4da1b970246c64797a91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:33.250375 5162 tablet_service.cc:1467] Processing CreateTablet for tablet 4c29ea3b807d4da1b970246c64797a91 (DEFAULT_TABLE table=rebalance_test_table_3 [id=23044fbda733476b84ec4cacff9be572]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:33.250788 5162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c29ea3b807d4da1b970246c64797a91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:33.252488 5290 tablet_service.cc:1467] Processing CreateTablet for tablet a3445b99b36444bb811486be77100479 (DEFAULT_TABLE table=rebalance_test_table_3 [id=23044fbda733476b84ec4cacff9be572]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:33.253078 5290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3445b99b36444bb811486be77100479. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:33.254436 5291 tablet_service.cc:1467] Processing CreateTablet for tablet 4c29ea3b807d4da1b970246c64797a91 (DEFAULT_TABLE table=rebalance_test_table_3 [id=23044fbda733476b84ec4cacff9be572]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:33.254952 5291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c29ea3b807d4da1b970246c64797a91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:33.255268 6311 tablet_bootstrap.cc:492] T 51b04d461eed462fa3660c27d75d05b7 P 9808124b81e14d1290ab0a6641a4b836: Bootstrap starting.
I20240501 08:13:33.257269 6311 tablet_bootstrap.cc:654] T 51b04d461eed462fa3660c27d75d05b7 P 9808124b81e14d1290ab0a6641a4b836: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:33.258076 6310 tablet_bootstrap.cc:492] T 51b04d461eed462fa3660c27d75d05b7 P b6cc443a494e4595b0e7cf1777620a3f: Bootstrap starting.
I20240501 08:13:33.260488 6310 tablet_bootstrap.cc:654] T 51b04d461eed462fa3660c27d75d05b7 P b6cc443a494e4595b0e7cf1777620a3f: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:33.263625 6314 tablet_bootstrap.cc:492] T 4c29ea3b807d4da1b970246c64797a91 P 0a952997a9fc4bfb8d7cbc56de6e0598: Bootstrap starting.
I20240501 08:13:33.266117 6314 tablet_bootstrap.cc:654] T 4c29ea3b807d4da1b970246c64797a91 P 0a952997a9fc4bfb8d7cbc56de6e0598: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:33.266680 6311 tablet_bootstrap.cc:492] T 51b04d461eed462fa3660c27d75d05b7 P 9808124b81e14d1290ab0a6641a4b836: No bootstrap required, opened a new log
I20240501 08:13:33.266847 6311 ts_tablet_manager.cc:1367] T 51b04d461eed462fa3660c27d75d05b7 P 9808124b81e14d1290ab0a6641a4b836: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20240501 08:13:33.267650 6311 raft_consensus.cc:348] T 51b04d461eed462fa3660c27d75d05b7 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:33.267870 6311 raft_consensus.cc:374] T 51b04d461eed462fa3660c27d75d05b7 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:33.267999 6311 raft_consensus.cc:729] T 51b04d461eed462fa3660c27d75d05b7 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Initialized, Role: FOLLOWER
I20240501 08:13:33.268282 6311 consensus_queue.cc:260] T 51b04d461eed462fa3660c27d75d05b7 P 9808124b81e14d1290ab0a6641a4b836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:33.268852 6311 ts_tablet_manager.cc:1398] T 51b04d461eed462fa3660c27d75d05b7 P 9808124b81e14d1290ab0a6641a4b836: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:13:33.270751 6314 tablet_bootstrap.cc:492] T 4c29ea3b807d4da1b970246c64797a91 P 0a952997a9fc4bfb8d7cbc56de6e0598: No bootstrap required, opened a new log
I20240501 08:13:33.270884 6314 ts_tablet_manager.cc:1367] T 4c29ea3b807d4da1b970246c64797a91 P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20240501 08:13:33.271625 6314 raft_consensus.cc:348] T 4c29ea3b807d4da1b970246c64797a91 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:33.271831 6314 raft_consensus.cc:374] T 4c29ea3b807d4da1b970246c64797a91 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:33.271929 6314 raft_consensus.cc:729] T 4c29ea3b807d4da1b970246c64797a91 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Initialized, Role: FOLLOWER
I20240501 08:13:33.272166 6314 consensus_queue.cc:260] T 4c29ea3b807d4da1b970246c64797a91 P 0a952997a9fc4bfb8d7cbc56de6e0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:33.274223 6314 ts_tablet_manager.cc:1398] T 4c29ea3b807d4da1b970246c64797a91 P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:13:33.274454 6310 tablet_bootstrap.cc:492] T 51b04d461eed462fa3660c27d75d05b7 P b6cc443a494e4595b0e7cf1777620a3f: No bootstrap required, opened a new log
I20240501 08:13:33.274528 6314 tablet_bootstrap.cc:492] T 51b04d461eed462fa3660c27d75d05b7 P 0a952997a9fc4bfb8d7cbc56de6e0598: Bootstrap starting.
I20240501 08:13:33.274613 6310 ts_tablet_manager.cc:1367] T 51b04d461eed462fa3660c27d75d05b7 P b6cc443a494e4595b0e7cf1777620a3f: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.005s
I20240501 08:13:33.275506 6310 raft_consensus.cc:348] T 51b04d461eed462fa3660c27d75d05b7 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:33.275755 6310 raft_consensus.cc:374] T 51b04d461eed462fa3660c27d75d05b7 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:33.275888 6310 raft_consensus.cc:729] T 51b04d461eed462fa3660c27d75d05b7 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6cc443a494e4595b0e7cf1777620a3f, State: Initialized, Role: FOLLOWER
I20240501 08:13:33.276293 6314 tablet_bootstrap.cc:654] T 51b04d461eed462fa3660c27d75d05b7 P 0a952997a9fc4bfb8d7cbc56de6e0598: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:33.276160 6310 consensus_queue.cc:260] T 51b04d461eed462fa3660c27d75d05b7 P b6cc443a494e4595b0e7cf1777620a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:33.276834 6310 ts_tablet_manager.cc:1398] T 51b04d461eed462fa3660c27d75d05b7 P b6cc443a494e4595b0e7cf1777620a3f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:33.277192 6310 tablet_bootstrap.cc:492] T a3445b99b36444bb811486be77100479 P b6cc443a494e4595b0e7cf1777620a3f: Bootstrap starting.
I20240501 08:13:33.277817 6311 tablet_bootstrap.cc:492] T 4c29ea3b807d4da1b970246c64797a91 P 9808124b81e14d1290ab0a6641a4b836: Bootstrap starting.
I20240501 08:13:33.279153 6310 tablet_bootstrap.cc:654] T a3445b99b36444bb811486be77100479 P b6cc443a494e4595b0e7cf1777620a3f: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:33.279332 6311 tablet_bootstrap.cc:654] T 4c29ea3b807d4da1b970246c64797a91 P 9808124b81e14d1290ab0a6641a4b836: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:33.281512 6311 tablet_bootstrap.cc:492] T 4c29ea3b807d4da1b970246c64797a91 P 9808124b81e14d1290ab0a6641a4b836: No bootstrap required, opened a new log
I20240501 08:13:33.281663 6311 ts_tablet_manager.cc:1367] T 4c29ea3b807d4da1b970246c64797a91 P 9808124b81e14d1290ab0a6641a4b836: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:13:33.282498 6311 raft_consensus.cc:348] T 4c29ea3b807d4da1b970246c64797a91 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:33.282878 6311 raft_consensus.cc:374] T 4c29ea3b807d4da1b970246c64797a91 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:33.283008 6311 raft_consensus.cc:729] T 4c29ea3b807d4da1b970246c64797a91 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Initialized, Role: FOLLOWER
I20240501 08:13:33.283390 6311 consensus_queue.cc:260] T 4c29ea3b807d4da1b970246c64797a91 P 9808124b81e14d1290ab0a6641a4b836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:33.284081 6311 ts_tablet_manager.cc:1398] T 4c29ea3b807d4da1b970246c64797a91 P 9808124b81e14d1290ab0a6641a4b836: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:33.284443 6311 tablet_bootstrap.cc:492] T a3445b99b36444bb811486be77100479 P 9808124b81e14d1290ab0a6641a4b836: Bootstrap starting.
I20240501 08:13:33.286033 6316 tablet_bootstrap.cc:492] T a3445b99b36444bb811486be77100479 P 2c4d718738d1408eabce489f4e140af6: Bootstrap starting.
I20240501 08:13:33.286140 6311 tablet_bootstrap.cc:654] T a3445b99b36444bb811486be77100479 P 9808124b81e14d1290ab0a6641a4b836: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:33.290810 6316 tablet_bootstrap.cc:654] T a3445b99b36444bb811486be77100479 P 2c4d718738d1408eabce489f4e140af6: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:33.291159 6314 tablet_bootstrap.cc:492] T 51b04d461eed462fa3660c27d75d05b7 P 0a952997a9fc4bfb8d7cbc56de6e0598: No bootstrap required, opened a new log
I20240501 08:13:33.291394 6314 ts_tablet_manager.cc:1367] T 51b04d461eed462fa3660c27d75d05b7 P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.001s
I20240501 08:13:33.292481 6314 raft_consensus.cc:348] T 51b04d461eed462fa3660c27d75d05b7 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:33.292820 6314 raft_consensus.cc:374] T 51b04d461eed462fa3660c27d75d05b7 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:33.292927 6314 raft_consensus.cc:729] T 51b04d461eed462fa3660c27d75d05b7 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Initialized, Role: FOLLOWER
I20240501 08:13:33.293279 6314 consensus_queue.cc:260] T 51b04d461eed462fa3660c27d75d05b7 P 0a952997a9fc4bfb8d7cbc56de6e0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:33.295083 6314 ts_tablet_manager.cc:1398] T 51b04d461eed462fa3660c27d75d05b7 P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240501 08:13:33.296813 6310 tablet_bootstrap.cc:492] T a3445b99b36444bb811486be77100479 P b6cc443a494e4595b0e7cf1777620a3f: No bootstrap required, opened a new log
I20240501 08:13:33.297039 6310 ts_tablet_manager.cc:1367] T a3445b99b36444bb811486be77100479 P b6cc443a494e4595b0e7cf1777620a3f: Time spent bootstrapping tablet: real 0.020s user 0.015s sys 0.003s
I20240501 08:13:33.298507 6310 raft_consensus.cc:348] T a3445b99b36444bb811486be77100479 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:33.298820 6310 raft_consensus.cc:374] T a3445b99b36444bb811486be77100479 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:33.298903 6310 raft_consensus.cc:729] T a3445b99b36444bb811486be77100479 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6cc443a494e4595b0e7cf1777620a3f, State: Initialized, Role: FOLLOWER
I20240501 08:13:33.299216 6311 tablet_bootstrap.cc:492] T a3445b99b36444bb811486be77100479 P 9808124b81e14d1290ab0a6641a4b836: No bootstrap required, opened a new log
I20240501 08:13:33.299276 6310 consensus_queue.cc:260] T a3445b99b36444bb811486be77100479 P b6cc443a494e4595b0e7cf1777620a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:33.299425 6311 ts_tablet_manager.cc:1367] T a3445b99b36444bb811486be77100479 P 9808124b81e14d1290ab0a6641a4b836: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.010s
I20240501 08:13:33.300060 6310 ts_tablet_manager.cc:1398] T a3445b99b36444bb811486be77100479 P b6cc443a494e4595b0e7cf1777620a3f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:13:33.300915 6316 tablet_bootstrap.cc:492] T a3445b99b36444bb811486be77100479 P 2c4d718738d1408eabce489f4e140af6: No bootstrap required, opened a new log
I20240501 08:13:33.301110 6316 ts_tablet_manager.cc:1367] T a3445b99b36444bb811486be77100479 P 2c4d718738d1408eabce489f4e140af6: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20240501 08:13:33.300992 6311 raft_consensus.cc:348] T a3445b99b36444bb811486be77100479 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:33.301209 6311 raft_consensus.cc:374] T a3445b99b36444bb811486be77100479 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:33.301307 6311 raft_consensus.cc:729] T a3445b99b36444bb811486be77100479 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Initialized, Role: FOLLOWER
I20240501 08:13:33.301896 6316 raft_consensus.cc:348] T a3445b99b36444bb811486be77100479 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:33.302150 6316 raft_consensus.cc:374] T a3445b99b36444bb811486be77100479 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:33.302275 6316 raft_consensus.cc:729] T a3445b99b36444bb811486be77100479 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Initialized, Role: FOLLOWER
I20240501 08:13:33.301541 6311 consensus_queue.cc:260] T a3445b99b36444bb811486be77100479 P 9808124b81e14d1290ab0a6641a4b836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:33.302505 6316 consensus_queue.cc:260] T a3445b99b36444bb811486be77100479 P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:33.303058 6316 ts_tablet_manager.cc:1398] T a3445b99b36444bb811486be77100479 P 2c4d718738d1408eabce489f4e140af6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:33.303362 6316 tablet_bootstrap.cc:492] T 4c29ea3b807d4da1b970246c64797a91 P 2c4d718738d1408eabce489f4e140af6: Bootstrap starting.
I20240501 08:13:33.305047 6316 tablet_bootstrap.cc:654] T 4c29ea3b807d4da1b970246c64797a91 P 2c4d718738d1408eabce489f4e140af6: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:33.306283 6311 ts_tablet_manager.cc:1398] T a3445b99b36444bb811486be77100479 P 9808124b81e14d1290ab0a6641a4b836: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20240501 08:13:33.307621 6316 tablet_bootstrap.cc:492] T 4c29ea3b807d4da1b970246c64797a91 P 2c4d718738d1408eabce489f4e140af6: No bootstrap required, opened a new log
I20240501 08:13:33.307768 6316 ts_tablet_manager.cc:1367] T 4c29ea3b807d4da1b970246c64797a91 P 2c4d718738d1408eabce489f4e140af6: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 08:13:33.308533 6316 raft_consensus.cc:348] T 4c29ea3b807d4da1b970246c64797a91 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:33.308737 6316 raft_consensus.cc:374] T 4c29ea3b807d4da1b970246c64797a91 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:33.308811 6316 raft_consensus.cc:729] T 4c29ea3b807d4da1b970246c64797a91 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Initialized, Role: FOLLOWER
I20240501 08:13:33.309022 6316 consensus_queue.cc:260] T 4c29ea3b807d4da1b970246c64797a91 P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:33.309470 6316 ts_tablet_manager.cc:1398] T 4c29ea3b807d4da1b970246c64797a91 P 2c4d718738d1408eabce489f4e140af6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:13:33.376109 6106 raft_consensus.cc:482] T 51b04d461eed462fa3660c27d75d05b7 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:33.376309 6106 raft_consensus.cc:504] T 51b04d461eed462fa3660c27d75d05b7 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:33.376869 6106 leader_election.cc:288] T 51b04d461eed462fa3660c27d75d05b7 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417), 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937)
I20240501 08:13:33.377444 5181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b04d461eed462fa3660c27d75d05b7" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" is_pre_election: true
I20240501 08:13:33.377450 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b04d461eed462fa3660c27d75d05b7" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f" is_pre_election: true
I20240501 08:13:33.377699 5181 raft_consensus.cc:2454] T 51b04d461eed462fa3660c27d75d05b7 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 0.
I20240501 08:13:33.377698 5051 raft_consensus.cc:2454] T 51b04d461eed462fa3660c27d75d05b7 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 0.
I20240501 08:13:33.378218 5247 leader_election.cc:302] T 51b04d461eed462fa3660c27d75d05b7 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9808124b81e14d1290ab0a6641a4b836, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:33.378506 6106 raft_consensus.cc:2781] T 51b04d461eed462fa3660c27d75d05b7 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:33.378602 6106 raft_consensus.cc:482] T 51b04d461eed462fa3660c27d75d05b7 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:33.378686 6106 raft_consensus.cc:3037] T 51b04d461eed462fa3660c27d75d05b7 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:33.380921 6106 raft_consensus.cc:504] T 51b04d461eed462fa3660c27d75d05b7 P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:33.381690 6106 leader_election.cc:288] T 51b04d461eed462fa3660c27d75d05b7 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 1 election: Requested vote from peers b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417), 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937)
I20240501 08:13:33.382193 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b04d461eed462fa3660c27d75d05b7" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:33.382193 5181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b04d461eed462fa3660c27d75d05b7" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:33.382419 5181 raft_consensus.cc:3037] T 51b04d461eed462fa3660c27d75d05b7 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:33.382419 5051 raft_consensus.cc:3037] T 51b04d461eed462fa3660c27d75d05b7 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:33.385228 5051 raft_consensus.cc:2454] T 51b04d461eed462fa3660c27d75d05b7 P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 1.
I20240501 08:13:33.385228 5181 raft_consensus.cc:2454] T 51b04d461eed462fa3660c27d75d05b7 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 1.
I20240501 08:13:33.385720 5247 leader_election.cc:302] T 51b04d461eed462fa3660c27d75d05b7 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9808124b81e14d1290ab0a6641a4b836, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:33.386098 6106 raft_consensus.cc:2781] T 51b04d461eed462fa3660c27d75d05b7 P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:33.386623 6106 raft_consensus.cc:686] T 51b04d461eed462fa3660c27d75d05b7 P 9808124b81e14d1290ab0a6641a4b836 [term 1 LEADER]: Becoming Leader. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:33.387032 6106 consensus_queue.cc:237] T 51b04d461eed462fa3660c27d75d05b7 P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:33.390420 4902 catalog_manager.cc:5484] T 51b04d461eed462fa3660c27d75d05b7 P 9808124b81e14d1290ab0a6641a4b836 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131). New cstate: current_term: 1 leader_uuid: "9808124b81e14d1290ab0a6641a4b836" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:33.423681 6290 raft_consensus.cc:482] T 4c29ea3b807d4da1b970246c64797a91 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:33.423861 6290 raft_consensus.cc:504] T 4c29ea3b807d4da1b970246c64797a91 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:33.424552 6290 leader_election.cc:288] T 4c29ea3b807d4da1b970246c64797a91 P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937), 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983)
I20240501 08:13:33.425024 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c29ea3b807d4da1b970246c64797a91" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808124b81e14d1290ab0a6641a4b836" is_pre_election: true
I20240501 08:13:33.425024 5181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c29ea3b807d4da1b970246c64797a91" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" is_pre_election: true
I20240501 08:13:33.425262 5311 raft_consensus.cc:2454] T 4c29ea3b807d4da1b970246c64797a91 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 0.
I20240501 08:13:33.425263 5181 raft_consensus.cc:2454] T 4c29ea3b807d4da1b970246c64797a91 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 0.
I20240501 08:13:33.425657 5377 leader_election.cc:302] T 4c29ea3b807d4da1b970246c64797a91 P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4d718738d1408eabce489f4e140af6, 9808124b81e14d1290ab0a6641a4b836; no voters:
I20240501 08:13:33.425915 6290 raft_consensus.cc:2781] T 4c29ea3b807d4da1b970246c64797a91 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:33.426065 6290 raft_consensus.cc:482] T 4c29ea3b807d4da1b970246c64797a91 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:33.426141 6290 raft_consensus.cc:3037] T 4c29ea3b807d4da1b970246c64797a91 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:33.428575 6290 raft_consensus.cc:504] T 4c29ea3b807d4da1b970246c64797a91 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:33.429134 6290 leader_election.cc:288] T 4c29ea3b807d4da1b970246c64797a91 P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 1 election: Requested vote from peers 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937), 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983)
I20240501 08:13:33.429805 5181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c29ea3b807d4da1b970246c64797a91" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:33.430069 5181 raft_consensus.cc:3037] T 4c29ea3b807d4da1b970246c64797a91 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:33.430073 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c29ea3b807d4da1b970246c64797a91" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
I20240501 08:13:33.430338 5311 raft_consensus.cc:3037] T 4c29ea3b807d4da1b970246c64797a91 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:33.432703 5181 raft_consensus.cc:2454] T 4c29ea3b807d4da1b970246c64797a91 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 1.
I20240501 08:13:33.433110 5376 leader_election.cc:302] T 4c29ea3b807d4da1b970246c64797a91 P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 2c4d718738d1408eabce489f4e140af6; no voters:
I20240501 08:13:33.433346 5311 raft_consensus.cc:2454] T 4c29ea3b807d4da1b970246c64797a91 P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 1.
I20240501 08:13:33.433403 6290 raft_consensus.cc:2781] T 4c29ea3b807d4da1b970246c64797a91 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:33.433503 6290 raft_consensus.cc:686] T 4c29ea3b807d4da1b970246c64797a91 P 2c4d718738d1408eabce489f4e140af6 [term 1 LEADER]: Becoming Leader. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:33.433773 6290 consensus_queue.cc:237] T 4c29ea3b807d4da1b970246c64797a91 P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:33.436940 4902 catalog_manager.cc:5484] T 4c29ea3b807d4da1b970246c64797a91 P 2c4d718738d1408eabce489f4e140af6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c4d718738d1408eabce489f4e140af6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "2c4d718738d1408eabce489f4e140af6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:33.525931 6179 raft_consensus.cc:482] T a3445b99b36444bb811486be77100479 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:33.526204 6179 raft_consensus.cc:504] T a3445b99b36444bb811486be77100479 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:33.526974 6179 leader_election.cc:288] T a3445b99b36444bb811486be77100479 P b6cc443a494e4595b0e7cf1777620a3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797), 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983)
I20240501 08:13:33.527413 5441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3445b99b36444bb811486be77100479" candidate_uuid: "b6cc443a494e4595b0e7cf1777620a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4d718738d1408eabce489f4e140af6" is_pre_election: true
I20240501 08:13:33.527578 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3445b99b36444bb811486be77100479" candidate_uuid: "b6cc443a494e4595b0e7cf1777620a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808124b81e14d1290ab0a6641a4b836" is_pre_election: true
I20240501 08:13:33.527671 5441 raft_consensus.cc:2454] T a3445b99b36444bb811486be77100479 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6cc443a494e4595b0e7cf1777620a3f in term 0.
I20240501 08:13:33.527825 5311 raft_consensus.cc:2454] T a3445b99b36444bb811486be77100479 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6cc443a494e4595b0e7cf1777620a3f in term 0.
I20240501 08:13:33.528095 4987 leader_election.cc:302] T a3445b99b36444bb811486be77100479 P b6cc443a494e4595b0e7cf1777620a3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4d718738d1408eabce489f4e140af6, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:33.528443 6179 raft_consensus.cc:2781] T a3445b99b36444bb811486be77100479 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:33.528558 6179 raft_consensus.cc:482] T a3445b99b36444bb811486be77100479 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:33.528643 6179 raft_consensus.cc:3037] T a3445b99b36444bb811486be77100479 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:33.530597 6179 raft_consensus.cc:504] T a3445b99b36444bb811486be77100479 P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:33.531291 6179 leader_election.cc:288] T a3445b99b36444bb811486be77100479 P b6cc443a494e4595b0e7cf1777620a3f [CANDIDATE]: Term 1 election: Requested vote from peers 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797), 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983)
I20240501 08:13:33.531733 5441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3445b99b36444bb811486be77100479" candidate_uuid: "b6cc443a494e4595b0e7cf1777620a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:33.531945 5441 raft_consensus.cc:3037] T a3445b99b36444bb811486be77100479 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:33.532356 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3445b99b36444bb811486be77100479" candidate_uuid: "b6cc443a494e4595b0e7cf1777620a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
I20240501 08:13:33.532565 5311 raft_consensus.cc:3037] T a3445b99b36444bb811486be77100479 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:33.534154 5441 raft_consensus.cc:2454] T a3445b99b36444bb811486be77100479 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6cc443a494e4595b0e7cf1777620a3f in term 1.
I20240501 08:13:33.534559 4987 leader_election.cc:302] T a3445b99b36444bb811486be77100479 P b6cc443a494e4595b0e7cf1777620a3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4d718738d1408eabce489f4e140af6, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:33.534816 6179 raft_consensus.cc:2781] T a3445b99b36444bb811486be77100479 P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:33.534932 6179 raft_consensus.cc:686] T a3445b99b36444bb811486be77100479 P b6cc443a494e4595b0e7cf1777620a3f [term 1 LEADER]: Becoming Leader. State: Replica: b6cc443a494e4595b0e7cf1777620a3f, State: Running, Role: LEADER
I20240501 08:13:33.535041 5311 raft_consensus.cc:2454] T a3445b99b36444bb811486be77100479 P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6cc443a494e4595b0e7cf1777620a3f in term 1.
I20240501 08:13:33.535200 6179 consensus_queue.cc:237] T a3445b99b36444bb811486be77100479 P b6cc443a494e4595b0e7cf1777620a3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:33.537799 4902 catalog_manager.cc:5484] T a3445b99b36444bb811486be77100479 P b6cc443a494e4595b0e7cf1777620a3f reported cstate change: term changed from 0 to 1, leader changed from <none> to b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129). New cstate: current_term: 1 leader_uuid: "b6cc443a494e4595b0e7cf1777620a3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:33.800246 6106 consensus_queue.cc:1035] T 51b04d461eed462fa3660c27d75d05b7 P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:33.818781 6323 consensus_queue.cc:1035] T 51b04d461eed462fa3660c27d75d05b7 P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:33.840862 6299 consensus_queue.cc:1035] T 4c29ea3b807d4da1b970246c64797a91 P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:33.848426 6336 consensus_queue.cc:1035] T 4c29ea3b807d4da1b970246c64797a91 P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:13:33.925271 6327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:33.925652 6327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 08:13:33.927713 6179 consensus_queue.cc:1035] T a3445b99b36444bb811486be77100479 P b6cc443a494e4595b0e7cf1777620a3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:33.932871 6262 consensus_queue.cc:1035] T a3445b99b36444bb811486be77100479 P b6cc443a494e4595b0e7cf1777620a3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:13:33.936203 6327 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 219.039 ms
time per row: 0.109519 ms
I20240501 08:13:34.305265 4903 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33308:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:13:34.314700 5290 tablet_service.cc:1467] Processing CreateTablet for tablet c5de53e441514804b45ab7d1c30edfa5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=13d75a245de8479e8a7208ae70a53feb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:34.314935 5162 tablet_service.cc:1467] Processing CreateTablet for tablet c5de53e441514804b45ab7d1c30edfa5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=13d75a245de8479e8a7208ae70a53feb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:34.315254 5290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5de53e441514804b45ab7d1c30edfa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:34.315399 5162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5de53e441514804b45ab7d1c30edfa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:34.315378 5029 tablet_service.cc:1467] Processing CreateTablet for tablet c5de53e441514804b45ab7d1c30edfa5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=13d75a245de8479e8a7208ae70a53feb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:34.315619 5161 tablet_service.cc:1467] Processing CreateTablet for tablet 98e5387cc69b483796e42cbf54f2226d (DEFAULT_TABLE table=rebalance_test_table_4 [id=13d75a245de8479e8a7208ae70a53feb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:34.315833 5029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5de53e441514804b45ab7d1c30edfa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:34.316035 5161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98e5387cc69b483796e42cbf54f2226d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:34.318919 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 98e5387cc69b483796e42cbf54f2226d (DEFAULT_TABLE table=rebalance_test_table_4 [id=13d75a245de8479e8a7208ae70a53feb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:34.319357 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98e5387cc69b483796e42cbf54f2226d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:34.324152 5420 tablet_service.cc:1467] Processing CreateTablet for tablet 0ad7f7e55fd34688b5eba16555306bc9 (DEFAULT_TABLE table=rebalance_test_table_4 [id=13d75a245de8479e8a7208ae70a53feb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:34.324690 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ad7f7e55fd34688b5eba16555306bc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:34.325215 5291 tablet_service.cc:1467] Processing CreateTablet for tablet 0ad7f7e55fd34688b5eba16555306bc9 (DEFAULT_TABLE table=rebalance_test_table_4 [id=13d75a245de8479e8a7208ae70a53feb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:34.325774 5291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ad7f7e55fd34688b5eba16555306bc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:34.330837 5030 tablet_service.cc:1467] Processing CreateTablet for tablet 98e5387cc69b483796e42cbf54f2226d (DEFAULT_TABLE table=rebalance_test_table_4 [id=13d75a245de8479e8a7208ae70a53feb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:34.331328 5030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98e5387cc69b483796e42cbf54f2226d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:34.334507 5031 tablet_service.cc:1467] Processing CreateTablet for tablet 0ad7f7e55fd34688b5eba16555306bc9 (DEFAULT_TABLE table=rebalance_test_table_4 [id=13d75a245de8479e8a7208ae70a53feb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:34.335029 5031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ad7f7e55fd34688b5eba16555306bc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:34.337504 6394 tablet_bootstrap.cc:492] T c5de53e441514804b45ab7d1c30edfa5 P 9808124b81e14d1290ab0a6641a4b836: Bootstrap starting.
I20240501 08:13:34.337643 6393 tablet_bootstrap.cc:492] T c5de53e441514804b45ab7d1c30edfa5 P 0a952997a9fc4bfb8d7cbc56de6e0598: Bootstrap starting.
I20240501 08:13:34.339468 6394 tablet_bootstrap.cc:654] T c5de53e441514804b45ab7d1c30edfa5 P 9808124b81e14d1290ab0a6641a4b836: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:34.339468 6393 tablet_bootstrap.cc:654] T c5de53e441514804b45ab7d1c30edfa5 P 0a952997a9fc4bfb8d7cbc56de6e0598: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:34.359994 6395 tablet_bootstrap.cc:492] T 98e5387cc69b483796e42cbf54f2226d P 2c4d718738d1408eabce489f4e140af6: Bootstrap starting.
I20240501 08:13:34.360213 6394 tablet_bootstrap.cc:492] T c5de53e441514804b45ab7d1c30edfa5 P 9808124b81e14d1290ab0a6641a4b836: No bootstrap required, opened a new log
I20240501 08:13:34.360365 6396 tablet_bootstrap.cc:492] T c5de53e441514804b45ab7d1c30edfa5 P b6cc443a494e4595b0e7cf1777620a3f: Bootstrap starting.
I20240501 08:13:34.360486 6394 ts_tablet_manager.cc:1367] T c5de53e441514804b45ab7d1c30edfa5 P 9808124b81e14d1290ab0a6641a4b836: Time spent bootstrapping tablet: real 0.023s user 0.015s sys 0.007s
I20240501 08:13:34.360580 6393 tablet_bootstrap.cc:492] T c5de53e441514804b45ab7d1c30edfa5 P 0a952997a9fc4bfb8d7cbc56de6e0598: No bootstrap required, opened a new log
I20240501 08:13:34.360822 6393 ts_tablet_manager.cc:1367] T c5de53e441514804b45ab7d1c30edfa5 P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.011s
I20240501 08:13:34.361428 6394 raft_consensus.cc:348] T c5de53e441514804b45ab7d1c30edfa5 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:34.361748 6394 raft_consensus.cc:374] T c5de53e441514804b45ab7d1c30edfa5 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:34.361732 6393 raft_consensus.cc:348] T c5de53e441514804b45ab7d1c30edfa5 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:34.361896 6394 raft_consensus.cc:729] T c5de53e441514804b45ab7d1c30edfa5 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Initialized, Role: FOLLOWER
I20240501 08:13:34.362040 6395 tablet_bootstrap.cc:654] T 98e5387cc69b483796e42cbf54f2226d P 2c4d718738d1408eabce489f4e140af6: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:34.362038 6393 raft_consensus.cc:374] T c5de53e441514804b45ab7d1c30edfa5 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:34.362138 6393 raft_consensus.cc:729] T c5de53e441514804b45ab7d1c30edfa5 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Initialized, Role: FOLLOWER
I20240501 08:13:34.362438 6393 consensus_queue.cc:260] T c5de53e441514804b45ab7d1c30edfa5 P 0a952997a9fc4bfb8d7cbc56de6e0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:34.362399 6394 consensus_queue.cc:260] T c5de53e441514804b45ab7d1c30edfa5 P 9808124b81e14d1290ab0a6641a4b836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:34.363157 6393 ts_tablet_manager.cc:1398] T c5de53e441514804b45ab7d1c30edfa5 P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:13:34.363675 6393 tablet_bootstrap.cc:492] T 98e5387cc69b483796e42cbf54f2226d P 0a952997a9fc4bfb8d7cbc56de6e0598: Bootstrap starting.
I20240501 08:13:34.366158 6393 tablet_bootstrap.cc:654] T 98e5387cc69b483796e42cbf54f2226d P 0a952997a9fc4bfb8d7cbc56de6e0598: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:34.366509 6394 ts_tablet_manager.cc:1398] T c5de53e441514804b45ab7d1c30edfa5 P 9808124b81e14d1290ab0a6641a4b836: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:13:34.367550 6394 tablet_bootstrap.cc:492] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836: Bootstrap starting.
I20240501 08:13:34.370432 6394 tablet_bootstrap.cc:654] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:34.373325 6396 tablet_bootstrap.cc:654] T c5de53e441514804b45ab7d1c30edfa5 P b6cc443a494e4595b0e7cf1777620a3f: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:34.376715 6393 tablet_bootstrap.cc:492] T 98e5387cc69b483796e42cbf54f2226d P 0a952997a9fc4bfb8d7cbc56de6e0598: No bootstrap required, opened a new log
I20240501 08:13:34.376989 6393 ts_tablet_manager.cc:1367] T 98e5387cc69b483796e42cbf54f2226d P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.008s
I20240501 08:13:34.377692 6394 tablet_bootstrap.cc:492] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836: No bootstrap required, opened a new log
I20240501 08:13:34.377884 6394 ts_tablet_manager.cc:1367] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240501 08:13:34.378070 6396 tablet_bootstrap.cc:492] T c5de53e441514804b45ab7d1c30edfa5 P b6cc443a494e4595b0e7cf1777620a3f: No bootstrap required, opened a new log
I20240501 08:13:34.378383 6396 ts_tablet_manager.cc:1367] T c5de53e441514804b45ab7d1c30edfa5 P b6cc443a494e4595b0e7cf1777620a3f: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.003s
I20240501 08:13:34.378388 6393 raft_consensus.cc:348] T 98e5387cc69b483796e42cbf54f2226d P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:34.378755 6394 raft_consensus.cc:348] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:34.378931 6393 raft_consensus.cc:374] T 98e5387cc69b483796e42cbf54f2226d P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:34.379065 6394 raft_consensus.cc:374] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:34.379087 6393 raft_consensus.cc:729] T 98e5387cc69b483796e42cbf54f2226d P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Initialized, Role: FOLLOWER
I20240501 08:13:34.379213 6394 raft_consensus.cc:729] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Initialized, Role: FOLLOWER
I20240501 08:13:34.379328 6396 raft_consensus.cc:348] T c5de53e441514804b45ab7d1c30edfa5 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:34.379590 6396 raft_consensus.cc:374] T c5de53e441514804b45ab7d1c30edfa5 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:34.379566 6394 consensus_queue.cc:260] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:34.379544 6393 consensus_queue.cc:260] T 98e5387cc69b483796e42cbf54f2226d P 0a952997a9fc4bfb8d7cbc56de6e0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:34.379741 6396 raft_consensus.cc:729] T c5de53e441514804b45ab7d1c30edfa5 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6cc443a494e4595b0e7cf1777620a3f, State: Initialized, Role: FOLLOWER
I20240501 08:13:34.382031 6393 ts_tablet_manager.cc:1398] T 98e5387cc69b483796e42cbf54f2226d P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:13:34.383517 6394 ts_tablet_manager.cc:1398] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:13:34.385247 6396 consensus_queue.cc:260] T c5de53e441514804b45ab7d1c30edfa5 P b6cc443a494e4595b0e7cf1777620a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:34.386147 6396 ts_tablet_manager.cc:1398] T c5de53e441514804b45ab7d1c30edfa5 P b6cc443a494e4595b0e7cf1777620a3f: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 08:13:34.386544 6396 tablet_bootstrap.cc:492] T 98e5387cc69b483796e42cbf54f2226d P b6cc443a494e4595b0e7cf1777620a3f: Bootstrap starting.
I20240501 08:13:34.388708 6396 tablet_bootstrap.cc:654] T 98e5387cc69b483796e42cbf54f2226d P b6cc443a494e4595b0e7cf1777620a3f: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:34.391253 6395 tablet_bootstrap.cc:492] T 98e5387cc69b483796e42cbf54f2226d P 2c4d718738d1408eabce489f4e140af6: No bootstrap required, opened a new log
I20240501 08:13:34.391471 6395 ts_tablet_manager.cc:1367] T 98e5387cc69b483796e42cbf54f2226d P 2c4d718738d1408eabce489f4e140af6: Time spent bootstrapping tablet: real 0.032s user 0.015s sys 0.015s
I20240501 08:13:34.391754 6396 tablet_bootstrap.cc:492] T 98e5387cc69b483796e42cbf54f2226d P b6cc443a494e4595b0e7cf1777620a3f: No bootstrap required, opened a new log
I20240501 08:13:34.391918 6396 ts_tablet_manager.cc:1367] T 98e5387cc69b483796e42cbf54f2226d P b6cc443a494e4595b0e7cf1777620a3f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240501 08:13:34.392304 6395 raft_consensus.cc:348] T 98e5387cc69b483796e42cbf54f2226d P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:34.392540 6395 raft_consensus.cc:374] T 98e5387cc69b483796e42cbf54f2226d P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:34.392637 6395 raft_consensus.cc:729] T 98e5387cc69b483796e42cbf54f2226d P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Initialized, Role: FOLLOWER
I20240501 08:13:34.392572 6396 raft_consensus.cc:348] T 98e5387cc69b483796e42cbf54f2226d P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:34.392827 6396 raft_consensus.cc:374] T 98e5387cc69b483796e42cbf54f2226d P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:34.392922 6396 raft_consensus.cc:729] T 98e5387cc69b483796e42cbf54f2226d P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6cc443a494e4595b0e7cf1777620a3f, State: Initialized, Role: FOLLOWER
I20240501 08:13:34.392872 6395 consensus_queue.cc:260] T 98e5387cc69b483796e42cbf54f2226d P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:34.393200 6396 consensus_queue.cc:260] T 98e5387cc69b483796e42cbf54f2226d P b6cc443a494e4595b0e7cf1777620a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:34.394271 6396 ts_tablet_manager.cc:1398] T 98e5387cc69b483796e42cbf54f2226d P b6cc443a494e4595b0e7cf1777620a3f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:13:34.394594 6396 tablet_bootstrap.cc:492] T 0ad7f7e55fd34688b5eba16555306bc9 P b6cc443a494e4595b0e7cf1777620a3f: Bootstrap starting.
I20240501 08:13:34.395869 6395 ts_tablet_manager.cc:1398] T 98e5387cc69b483796e42cbf54f2226d P 2c4d718738d1408eabce489f4e140af6: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 08:13:34.396289 6396 tablet_bootstrap.cc:654] T 0ad7f7e55fd34688b5eba16555306bc9 P b6cc443a494e4595b0e7cf1777620a3f: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:34.397011 6395 tablet_bootstrap.cc:492] T 0ad7f7e55fd34688b5eba16555306bc9 P 2c4d718738d1408eabce489f4e140af6: Bootstrap starting.
I20240501 08:13:34.398586 6395 tablet_bootstrap.cc:654] T 0ad7f7e55fd34688b5eba16555306bc9 P 2c4d718738d1408eabce489f4e140af6: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:34.401217 6396 tablet_bootstrap.cc:492] T 0ad7f7e55fd34688b5eba16555306bc9 P b6cc443a494e4595b0e7cf1777620a3f: No bootstrap required, opened a new log
I20240501 08:13:34.401371 6396 ts_tablet_manager.cc:1367] T 0ad7f7e55fd34688b5eba16555306bc9 P b6cc443a494e4595b0e7cf1777620a3f: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20240501 08:13:34.401582 6395 tablet_bootstrap.cc:492] T 0ad7f7e55fd34688b5eba16555306bc9 P 2c4d718738d1408eabce489f4e140af6: No bootstrap required, opened a new log
I20240501 08:13:34.401739 6395 ts_tablet_manager.cc:1367] T 0ad7f7e55fd34688b5eba16555306bc9 P 2c4d718738d1408eabce489f4e140af6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 08:13:34.402091 6396 raft_consensus.cc:348] T 0ad7f7e55fd34688b5eba16555306bc9 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:34.402271 6396 raft_consensus.cc:374] T 0ad7f7e55fd34688b5eba16555306bc9 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:34.402393 6396 raft_consensus.cc:729] T 0ad7f7e55fd34688b5eba16555306bc9 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6cc443a494e4595b0e7cf1777620a3f, State: Initialized, Role: FOLLOWER
I20240501 08:13:34.402447 6395 raft_consensus.cc:348] T 0ad7f7e55fd34688b5eba16555306bc9 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:34.402670 6395 raft_consensus.cc:374] T 0ad7f7e55fd34688b5eba16555306bc9 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:34.402617 6396 consensus_queue.cc:260] T 0ad7f7e55fd34688b5eba16555306bc9 P b6cc443a494e4595b0e7cf1777620a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:34.402738 6395 raft_consensus.cc:729] T 0ad7f7e55fd34688b5eba16555306bc9 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Initialized, Role: FOLLOWER
I20240501 08:13:34.402976 6395 consensus_queue.cc:260] T 0ad7f7e55fd34688b5eba16555306bc9 P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:34.403278 6396 ts_tablet_manager.cc:1398] T 0ad7f7e55fd34688b5eba16555306bc9 P b6cc443a494e4595b0e7cf1777620a3f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:34.404170 6395 ts_tablet_manager.cc:1398] T 0ad7f7e55fd34688b5eba16555306bc9 P 2c4d718738d1408eabce489f4e140af6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:34.450286 6289 raft_consensus.cc:482] T 98e5387cc69b483796e42cbf54f2226d P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:34.450495 6289 raft_consensus.cc:504] T 98e5387cc69b483796e42cbf54f2226d P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:34.451133 6289 leader_election.cc:288] T 98e5387cc69b483796e42cbf54f2226d P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797), b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417)
I20240501 08:13:34.451666 5441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e5387cc69b483796e42cbf54f2226d" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4d718738d1408eabce489f4e140af6" is_pre_election: true
I20240501 08:13:34.451714 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e5387cc69b483796e42cbf54f2226d" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f" is_pre_election: true
I20240501 08:13:34.451864 5441 raft_consensus.cc:2454] T 98e5387cc69b483796e42cbf54f2226d P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 0.
I20240501 08:13:34.451970 5051 raft_consensus.cc:2454] T 98e5387cc69b483796e42cbf54f2226d P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 0.
I20240501 08:13:34.452286 5118 leader_election.cc:302] T 98e5387cc69b483796e42cbf54f2226d P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 2c4d718738d1408eabce489f4e140af6; no voters:
I20240501 08:13:34.452580 6289 raft_consensus.cc:2781] T 98e5387cc69b483796e42cbf54f2226d P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:34.452661 6289 raft_consensus.cc:482] T 98e5387cc69b483796e42cbf54f2226d P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:34.452759 6289 raft_consensus.cc:3037] T 98e5387cc69b483796e42cbf54f2226d P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:34.454710 6289 raft_consensus.cc:504] T 98e5387cc69b483796e42cbf54f2226d P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:34.455305 6289 leader_election.cc:288] T 98e5387cc69b483796e42cbf54f2226d P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 election: Requested vote from peers 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797), b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417)
I20240501 08:13:34.455713 5441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e5387cc69b483796e42cbf54f2226d" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:34.455767 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e5387cc69b483796e42cbf54f2226d" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:34.455894 5441 raft_consensus.cc:3037] T 98e5387cc69b483796e42cbf54f2226d P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:34.455991 5051 raft_consensus.cc:3037] T 98e5387cc69b483796e42cbf54f2226d P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:34.457901 5051 raft_consensus.cc:2454] T 98e5387cc69b483796e42cbf54f2226d P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 1.
I20240501 08:13:34.458072 5441 raft_consensus.cc:2454] T 98e5387cc69b483796e42cbf54f2226d P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 1.
I20240501 08:13:34.458518 5117 leader_election.cc:302] T 98e5387cc69b483796e42cbf54f2226d P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:34.458958 6289 raft_consensus.cc:2781] T 98e5387cc69b483796e42cbf54f2226d P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:34.459393 6289 raft_consensus.cc:686] T 98e5387cc69b483796e42cbf54f2226d P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Becoming Leader. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Running, Role: LEADER
I20240501 08:13:34.459913 6289 consensus_queue.cc:237] T 98e5387cc69b483796e42cbf54f2226d P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:34.462819 4903 catalog_manager.cc:5484] T 98e5387cc69b483796e42cbf54f2226d P 0a952997a9fc4bfb8d7cbc56de6e0598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130). New cstate: current_term: 1 leader_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:34.544225 6289 raft_consensus.cc:482] T c5de53e441514804b45ab7d1c30edfa5 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:34.544445 6289 raft_consensus.cc:504] T c5de53e441514804b45ab7d1c30edfa5 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:34.545071 6289 leader_election.cc:288] T c5de53e441514804b45ab7d1c30edfa5 P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983), b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417)
I20240501 08:13:34.545682 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5de53e441514804b45ab7d1c30edfa5" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808124b81e14d1290ab0a6641a4b836" is_pre_election: true
I20240501 08:13:34.545749 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5de53e441514804b45ab7d1c30edfa5" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f" is_pre_election: true
I20240501 08:13:34.545919 5311 raft_consensus.cc:2454] T c5de53e441514804b45ab7d1c30edfa5 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 0.
I20240501 08:13:34.546013 5051 raft_consensus.cc:2454] T c5de53e441514804b45ab7d1c30edfa5 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 0.
I20240501 08:13:34.546419 5117 leader_election.cc:302] T c5de53e441514804b45ab7d1c30edfa5 P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 9808124b81e14d1290ab0a6641a4b836; no voters:
I20240501 08:13:34.546675 6289 raft_consensus.cc:2781] T c5de53e441514804b45ab7d1c30edfa5 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:34.546773 6289 raft_consensus.cc:482] T c5de53e441514804b45ab7d1c30edfa5 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:34.546861 6289 raft_consensus.cc:3037] T c5de53e441514804b45ab7d1c30edfa5 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:34.549185 6289 raft_consensus.cc:504] T c5de53e441514804b45ab7d1c30edfa5 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:34.549989 6289 leader_election.cc:288] T c5de53e441514804b45ab7d1c30edfa5 P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 election: Requested vote from peers 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983), b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417)
I20240501 08:13:34.550535 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5de53e441514804b45ab7d1c30edfa5" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
I20240501 08:13:34.550618 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5de53e441514804b45ab7d1c30edfa5" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:34.550798 5311 raft_consensus.cc:3037] T c5de53e441514804b45ab7d1c30edfa5 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:34.550817 5051 raft_consensus.cc:3037] T c5de53e441514804b45ab7d1c30edfa5 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:34.553751 5311 raft_consensus.cc:2454] T c5de53e441514804b45ab7d1c30edfa5 P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 1.
I20240501 08:13:34.553787 5051 raft_consensus.cc:2454] T c5de53e441514804b45ab7d1c30edfa5 P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 1.
I20240501 08:13:34.554313 5117 leader_election.cc:302] T c5de53e441514804b45ab7d1c30edfa5 P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:34.554648 6289 raft_consensus.cc:2781] T c5de53e441514804b45ab7d1c30edfa5 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:34.554841 6289 raft_consensus.cc:686] T c5de53e441514804b45ab7d1c30edfa5 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Becoming Leader. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Running, Role: LEADER
I20240501 08:13:34.555418 6289 consensus_queue.cc:237] T c5de53e441514804b45ab7d1c30edfa5 P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:34.558485 4903 catalog_manager.cc:5484] T c5de53e441514804b45ab7d1c30edfa5 P 0a952997a9fc4bfb8d7cbc56de6e0598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130). New cstate: current_term: 1 leader_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:34.634702 6323 raft_consensus.cc:482] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:34.634918 6323 raft_consensus.cc:504] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:34.635685 6323 leader_election.cc:288] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797), b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417)
I20240501 08:13:34.636276 5441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ad7f7e55fd34688b5eba16555306bc9" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4d718738d1408eabce489f4e140af6" is_pre_election: true
I20240501 08:13:34.636313 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ad7f7e55fd34688b5eba16555306bc9" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f" is_pre_election: true
I20240501 08:13:34.636516 5441 raft_consensus.cc:2454] T 0ad7f7e55fd34688b5eba16555306bc9 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 0.
I20240501 08:13:34.636564 5051 raft_consensus.cc:2454] T 0ad7f7e55fd34688b5eba16555306bc9 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 0.
I20240501 08:13:34.637003 5248 leader_election.cc:302] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4d718738d1408eabce489f4e140af6, 9808124b81e14d1290ab0a6641a4b836; no voters:
I20240501 08:13:34.637310 6323 raft_consensus.cc:2781] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:34.637398 6323 raft_consensus.cc:482] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:34.637467 6323 raft_consensus.cc:3037] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:34.639377 6323 raft_consensus.cc:504] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:34.640026 6323 leader_election.cc:288] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 1 election: Requested vote from peers 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797), b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417)
I20240501 08:13:34.640530 5441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ad7f7e55fd34688b5eba16555306bc9" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:34.640614 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ad7f7e55fd34688b5eba16555306bc9" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:34.640750 5441 raft_consensus.cc:3037] T 0ad7f7e55fd34688b5eba16555306bc9 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:34.640856 5051 raft_consensus.cc:3037] T 0ad7f7e55fd34688b5eba16555306bc9 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:34.643087 5051 raft_consensus.cc:2454] T 0ad7f7e55fd34688b5eba16555306bc9 P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 1.
I20240501 08:13:34.643082 5441 raft_consensus.cc:2454] T 0ad7f7e55fd34688b5eba16555306bc9 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 1.
I20240501 08:13:34.643503 5248 leader_election.cc:302] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4d718738d1408eabce489f4e140af6, 9808124b81e14d1290ab0a6641a4b836; no voters:
I20240501 08:13:34.643836 6323 raft_consensus.cc:2781] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:34.644001 6323 raft_consensus.cc:686] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836 [term 1 LEADER]: Becoming Leader. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:34.644294 6323 consensus_queue.cc:237] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:34.647231 4903 catalog_manager.cc:5484] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131). New cstate: current_term: 1 leader_uuid: "9808124b81e14d1290ab0a6641a4b836" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:34.904145 6289 consensus_queue.cc:1035] T 98e5387cc69b483796e42cbf54f2226d P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:34.911016 6406 consensus_queue.cc:1035] T 98e5387cc69b483796e42cbf54f2226d P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:34.969781 6289 consensus_queue.cc:1035] T c5de53e441514804b45ab7d1c30edfa5 P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:34.971875 6289 consensus_queue.cc:1035] T c5de53e441514804b45ab7d1c30edfa5 P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:35.080739 6323 consensus_queue.cc:1035] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:13:35.083418 6410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:35.083870 6410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 08:13:35.085542 6106 consensus_queue.cc:1035] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:13:35.094147 6410 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 273.853 ms
time per row: 0.136927 ms
I20240501 08:13:35.511803 4907 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33308:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:13:35.521422 5290 tablet_service.cc:1467] Processing CreateTablet for tablet f518dc436d1d4e9dbe0ec9bbaf17e5fd (DEFAULT_TABLE table=rebalance_test_table_5 [id=97db063eb61e411599026379bfa986a5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:35.521667 5422 tablet_service.cc:1467] Processing CreateTablet for tablet f518dc436d1d4e9dbe0ec9bbaf17e5fd (DEFAULT_TABLE table=rebalance_test_table_5 [id=97db063eb61e411599026379bfa986a5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:35.522110 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f518dc436d1d4e9dbe0ec9bbaf17e5fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:35.522541 5420 tablet_service.cc:1467] Processing CreateTablet for tablet 97943d9ec11f4c5da7a33a91818d9978 (DEFAULT_TABLE table=rebalance_test_table_5 [id=97db063eb61e411599026379bfa986a5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:35.522789 5162 tablet_service.cc:1467] Processing CreateTablet for tablet f518dc436d1d4e9dbe0ec9bbaf17e5fd (DEFAULT_TABLE table=rebalance_test_table_5 [id=97db063eb61e411599026379bfa986a5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:35.522912 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97943d9ec11f4c5da7a33a91818d9978. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:35.523077 5161 tablet_service.cc:1467] Processing CreateTablet for tablet 97943d9ec11f4c5da7a33a91818d9978 (DEFAULT_TABLE table=rebalance_test_table_5 [id=97db063eb61e411599026379bfa986a5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:35.523198 5162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f518dc436d1d4e9dbe0ec9bbaf17e5fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:35.523460 5161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97943d9ec11f4c5da7a33a91818d9978. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:35.524510 5160 tablet_service.cc:1467] Processing CreateTablet for tablet e91cc538a8154588ac05a1e7e508b472 (DEFAULT_TABLE table=rebalance_test_table_5 [id=97db063eb61e411599026379bfa986a5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:35.524849 5160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e91cc538a8154588ac05a1e7e508b472. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:35.521934 5290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f518dc436d1d4e9dbe0ec9bbaf17e5fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:35.522715 5291 tablet_service.cc:1467] Processing CreateTablet for tablet 97943d9ec11f4c5da7a33a91818d9978 (DEFAULT_TABLE table=rebalance_test_table_5 [id=97db063eb61e411599026379bfa986a5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:35.530661 5291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97943d9ec11f4c5da7a33a91818d9978. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:35.536731 5029 tablet_service.cc:1467] Processing CreateTablet for tablet e91cc538a8154588ac05a1e7e508b472 (DEFAULT_TABLE table=rebalance_test_table_5 [id=97db063eb61e411599026379bfa986a5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:35.537056 5292 tablet_service.cc:1467] Processing CreateTablet for tablet e91cc538a8154588ac05a1e7e508b472 (DEFAULT_TABLE table=rebalance_test_table_5 [id=97db063eb61e411599026379bfa986a5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:35.537593 5292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e91cc538a8154588ac05a1e7e508b472. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:35.537207 5029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e91cc538a8154588ac05a1e7e508b472. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:35.543882 6479 tablet_bootstrap.cc:492] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 0a952997a9fc4bfb8d7cbc56de6e0598: Bootstrap starting.
I20240501 08:13:35.545874 6479 tablet_bootstrap.cc:654] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 0a952997a9fc4bfb8d7cbc56de6e0598: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:35.549180 6480 tablet_bootstrap.cc:492] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 9808124b81e14d1290ab0a6641a4b836: Bootstrap starting.
I20240501 08:13:35.551050 6480 tablet_bootstrap.cc:654] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 9808124b81e14d1290ab0a6641a4b836: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:35.552183 6479 tablet_bootstrap.cc:492] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 0a952997a9fc4bfb8d7cbc56de6e0598: No bootstrap required, opened a new log
I20240501 08:13:35.552377 6479 ts_tablet_manager.cc:1367] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240501 08:13:35.553031 6481 tablet_bootstrap.cc:492] T e91cc538a8154588ac05a1e7e508b472 P b6cc443a494e4595b0e7cf1777620a3f: Bootstrap starting.
I20240501 08:13:35.553254 6479 raft_consensus.cc:348] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:35.553553 6479 raft_consensus.cc:374] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:35.553675 6479 raft_consensus.cc:729] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Initialized, Role: FOLLOWER
I20240501 08:13:35.554071 6479 consensus_queue.cc:260] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 0a952997a9fc4bfb8d7cbc56de6e0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:35.555037 6479 ts_tablet_manager.cc:1398] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 08:13:35.556551 6481 tablet_bootstrap.cc:654] T e91cc538a8154588ac05a1e7e508b472 P b6cc443a494e4595b0e7cf1777620a3f: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:35.557902 6479 tablet_bootstrap.cc:492] T e91cc538a8154588ac05a1e7e508b472 P 0a952997a9fc4bfb8d7cbc56de6e0598: Bootstrap starting.
I20240501 08:13:35.558815 6478 tablet_bootstrap.cc:492] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 2c4d718738d1408eabce489f4e140af6: Bootstrap starting.
I20240501 08:13:35.560954 6479 tablet_bootstrap.cc:654] T e91cc538a8154588ac05a1e7e508b472 P 0a952997a9fc4bfb8d7cbc56de6e0598: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:35.562985 6478 tablet_bootstrap.cc:654] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 2c4d718738d1408eabce489f4e140af6: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:35.565467 6480 tablet_bootstrap.cc:492] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 9808124b81e14d1290ab0a6641a4b836: No bootstrap required, opened a new log
I20240501 08:13:35.565534 6481 tablet_bootstrap.cc:492] T e91cc538a8154588ac05a1e7e508b472 P b6cc443a494e4595b0e7cf1777620a3f: No bootstrap required, opened a new log
I20240501 08:13:35.565757 6480 ts_tablet_manager.cc:1367] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 9808124b81e14d1290ab0a6641a4b836: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.005s
I20240501 08:13:35.566941 6481 ts_tablet_manager.cc:1367] T e91cc538a8154588ac05a1e7e508b472 P b6cc443a494e4595b0e7cf1777620a3f: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20240501 08:13:35.567854 6481 raft_consensus.cc:348] T e91cc538a8154588ac05a1e7e508b472 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:35.568159 6481 raft_consensus.cc:374] T e91cc538a8154588ac05a1e7e508b472 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:35.568287 6481 raft_consensus.cc:729] T e91cc538a8154588ac05a1e7e508b472 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6cc443a494e4595b0e7cf1777620a3f, State: Initialized, Role: FOLLOWER
I20240501 08:13:35.568782 6481 consensus_queue.cc:260] T e91cc538a8154588ac05a1e7e508b472 P b6cc443a494e4595b0e7cf1777620a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:35.569346 6478 tablet_bootstrap.cc:492] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 2c4d718738d1408eabce489f4e140af6: No bootstrap required, opened a new log
I20240501 08:13:35.571172 6481 ts_tablet_manager.cc:1398] T e91cc538a8154588ac05a1e7e508b472 P b6cc443a494e4595b0e7cf1777620a3f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:13:35.571625 6478 ts_tablet_manager.cc:1367] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 2c4d718738d1408eabce489f4e140af6: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240501 08:13:35.572587 6480 raft_consensus.cc:348] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:35.572883 6480 raft_consensus.cc:374] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:35.572981 6480 raft_consensus.cc:729] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Initialized, Role: FOLLOWER
I20240501 08:13:35.573324 6480 consensus_queue.cc:260] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 9808124b81e14d1290ab0a6641a4b836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:35.573494 6478 raft_consensus.cc:348] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:35.574260 6480 ts_tablet_manager.cc:1398] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 9808124b81e14d1290ab0a6641a4b836: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240501 08:13:35.574620 6478 raft_consensus.cc:374] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:35.574792 6478 raft_consensus.cc:729] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Initialized, Role: FOLLOWER
I20240501 08:13:35.575114 6480 tablet_bootstrap.cc:492] T 97943d9ec11f4c5da7a33a91818d9978 P 9808124b81e14d1290ab0a6641a4b836: Bootstrap starting.
I20240501 08:13:35.575134 6478 consensus_queue.cc:260] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:35.577458 6480 tablet_bootstrap.cc:654] T 97943d9ec11f4c5da7a33a91818d9978 P 9808124b81e14d1290ab0a6641a4b836: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:35.578648 6478 ts_tablet_manager.cc:1398] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 2c4d718738d1408eabce489f4e140af6: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 08:13:35.581723 6480 tablet_bootstrap.cc:492] T 97943d9ec11f4c5da7a33a91818d9978 P 9808124b81e14d1290ab0a6641a4b836: No bootstrap required, opened a new log
I20240501 08:13:35.581969 6480 ts_tablet_manager.cc:1367] T 97943d9ec11f4c5da7a33a91818d9978 P 9808124b81e14d1290ab0a6641a4b836: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20240501 08:13:35.583184 6478 tablet_bootstrap.cc:492] T 97943d9ec11f4c5da7a33a91818d9978 P 2c4d718738d1408eabce489f4e140af6: Bootstrap starting.
I20240501 08:13:35.584770 6478 tablet_bootstrap.cc:654] T 97943d9ec11f4c5da7a33a91818d9978 P 2c4d718738d1408eabce489f4e140af6: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:35.583024 6480 raft_consensus.cc:348] T 97943d9ec11f4c5da7a33a91818d9978 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:35.585317 6480 raft_consensus.cc:374] T 97943d9ec11f4c5da7a33a91818d9978 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:35.585395 6480 raft_consensus.cc:729] T 97943d9ec11f4c5da7a33a91818d9978 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Initialized, Role: FOLLOWER
I20240501 08:13:35.585675 6480 consensus_queue.cc:260] T 97943d9ec11f4c5da7a33a91818d9978 P 9808124b81e14d1290ab0a6641a4b836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:35.586369 6480 ts_tablet_manager.cc:1398] T 97943d9ec11f4c5da7a33a91818d9978 P 9808124b81e14d1290ab0a6641a4b836: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:13:35.586511 6479 tablet_bootstrap.cc:492] T e91cc538a8154588ac05a1e7e508b472 P 0a952997a9fc4bfb8d7cbc56de6e0598: No bootstrap required, opened a new log
I20240501 08:13:35.586730 6479 ts_tablet_manager.cc:1367] T e91cc538a8154588ac05a1e7e508b472 P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent bootstrapping tablet: real 0.029s user 0.015s sys 0.011s
I20240501 08:13:35.587628 6479 raft_consensus.cc:348] T e91cc538a8154588ac05a1e7e508b472 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:35.587922 6479 raft_consensus.cc:374] T e91cc538a8154588ac05a1e7e508b472 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:35.588140 6479 raft_consensus.cc:729] T e91cc538a8154588ac05a1e7e508b472 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Initialized, Role: FOLLOWER
I20240501 08:13:35.588480 6479 consensus_queue.cc:260] T e91cc538a8154588ac05a1e7e508b472 P 0a952997a9fc4bfb8d7cbc56de6e0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:35.589097 6479 ts_tablet_manager.cc:1398] T e91cc538a8154588ac05a1e7e508b472 P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:35.589516 6479 tablet_bootstrap.cc:492] T 97943d9ec11f4c5da7a33a91818d9978 P 0a952997a9fc4bfb8d7cbc56de6e0598: Bootstrap starting.
I20240501 08:13:35.591229 6480 tablet_bootstrap.cc:492] T e91cc538a8154588ac05a1e7e508b472 P 9808124b81e14d1290ab0a6641a4b836: Bootstrap starting.
I20240501 08:13:35.591420 6479 tablet_bootstrap.cc:654] T 97943d9ec11f4c5da7a33a91818d9978 P 0a952997a9fc4bfb8d7cbc56de6e0598: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:35.594008 6480 tablet_bootstrap.cc:654] T e91cc538a8154588ac05a1e7e508b472 P 9808124b81e14d1290ab0a6641a4b836: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:35.598057 6478 tablet_bootstrap.cc:492] T 97943d9ec11f4c5da7a33a91818d9978 P 2c4d718738d1408eabce489f4e140af6: No bootstrap required, opened a new log
I20240501 08:13:35.598277 6478 ts_tablet_manager.cc:1367] T 97943d9ec11f4c5da7a33a91818d9978 P 2c4d718738d1408eabce489f4e140af6: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.005s
I20240501 08:13:35.599030 6478 raft_consensus.cc:348] T 97943d9ec11f4c5da7a33a91818d9978 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:35.599287 6478 raft_consensus.cc:374] T 97943d9ec11f4c5da7a33a91818d9978 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:35.599382 6478 raft_consensus.cc:729] T 97943d9ec11f4c5da7a33a91818d9978 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Initialized, Role: FOLLOWER
I20240501 08:13:35.599625 6478 consensus_queue.cc:260] T 97943d9ec11f4c5da7a33a91818d9978 P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:35.600198 6478 ts_tablet_manager.cc:1398] T 97943d9ec11f4c5da7a33a91818d9978 P 2c4d718738d1408eabce489f4e140af6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:13:35.601655 6480 tablet_bootstrap.cc:492] T e91cc538a8154588ac05a1e7e508b472 P 9808124b81e14d1290ab0a6641a4b836: No bootstrap required, opened a new log
I20240501 08:13:35.601814 6480 ts_tablet_manager.cc:1367] T e91cc538a8154588ac05a1e7e508b472 P 9808124b81e14d1290ab0a6641a4b836: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.003s
I20240501 08:13:35.602324 6479 tablet_bootstrap.cc:492] T 97943d9ec11f4c5da7a33a91818d9978 P 0a952997a9fc4bfb8d7cbc56de6e0598: No bootstrap required, opened a new log
I20240501 08:13:35.602476 6479 ts_tablet_manager.cc:1367] T 97943d9ec11f4c5da7a33a91818d9978 P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20240501 08:13:35.602687 6480 raft_consensus.cc:348] T e91cc538a8154588ac05a1e7e508b472 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:35.602917 6480 raft_consensus.cc:374] T e91cc538a8154588ac05a1e7e508b472 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:35.603024 6480 raft_consensus.cc:729] T e91cc538a8154588ac05a1e7e508b472 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Initialized, Role: FOLLOWER
I20240501 08:13:35.603253 6480 consensus_queue.cc:260] T e91cc538a8154588ac05a1e7e508b472 P 9808124b81e14d1290ab0a6641a4b836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:35.603248 6479 raft_consensus.cc:348] T 97943d9ec11f4c5da7a33a91818d9978 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:35.603463 6479 raft_consensus.cc:374] T 97943d9ec11f4c5da7a33a91818d9978 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:35.603557 6479 raft_consensus.cc:729] T 97943d9ec11f4c5da7a33a91818d9978 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Initialized, Role: FOLLOWER
I20240501 08:13:35.603806 6480 ts_tablet_manager.cc:1398] T e91cc538a8154588ac05a1e7e508b472 P 9808124b81e14d1290ab0a6641a4b836: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:13:35.603816 6479 consensus_queue.cc:260] T 97943d9ec11f4c5da7a33a91818d9978 P 0a952997a9fc4bfb8d7cbc56de6e0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:35.604374 6479 ts_tablet_manager.cc:1398] T 97943d9ec11f4c5da7a33a91818d9978 P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:35.624117 6406 raft_consensus.cc:482] T 97943d9ec11f4c5da7a33a91818d9978 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:35.624333 6406 raft_consensus.cc:504] T 97943d9ec11f4c5da7a33a91818d9978 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:35.624969 6406 leader_election.cc:288] T 97943d9ec11f4c5da7a33a91818d9978 P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983), 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797)
I20240501 08:13:35.625553 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97943d9ec11f4c5da7a33a91818d9978" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808124b81e14d1290ab0a6641a4b836" is_pre_election: true
I20240501 08:13:35.625572 5441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97943d9ec11f4c5da7a33a91818d9978" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4d718738d1408eabce489f4e140af6" is_pre_election: true
I20240501 08:13:35.625813 5441 raft_consensus.cc:2454] T 97943d9ec11f4c5da7a33a91818d9978 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 0.
I20240501 08:13:35.625800 5311 raft_consensus.cc:2454] T 97943d9ec11f4c5da7a33a91818d9978 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 0.
I20240501 08:13:35.626287 5117 leader_election.cc:302] T 97943d9ec11f4c5da7a33a91818d9978 P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 9808124b81e14d1290ab0a6641a4b836; no voters:
I20240501 08:13:35.626631 6406 raft_consensus.cc:2781] T 97943d9ec11f4c5da7a33a91818d9978 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:35.626737 6406 raft_consensus.cc:482] T 97943d9ec11f4c5da7a33a91818d9978 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:35.626827 6406 raft_consensus.cc:3037] T 97943d9ec11f4c5da7a33a91818d9978 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:35.629041 6406 raft_consensus.cc:504] T 97943d9ec11f4c5da7a33a91818d9978 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:35.629649 6406 leader_election.cc:288] T 97943d9ec11f4c5da7a33a91818d9978 P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 election: Requested vote from peers 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983), 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797)
I20240501 08:13:35.630122 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97943d9ec11f4c5da7a33a91818d9978" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
I20240501 08:13:35.630223 5441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97943d9ec11f4c5da7a33a91818d9978" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:35.630378 5311 raft_consensus.cc:3037] T 97943d9ec11f4c5da7a33a91818d9978 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:35.630409 5441 raft_consensus.cc:3037] T 97943d9ec11f4c5da7a33a91818d9978 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:35.633282 5311 raft_consensus.cc:2454] T 97943d9ec11f4c5da7a33a91818d9978 P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 1.
I20240501 08:13:35.633272 5441 raft_consensus.cc:2454] T 97943d9ec11f4c5da7a33a91818d9978 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 1.
I20240501 08:13:35.633755 5118 leader_election.cc:302] T 97943d9ec11f4c5da7a33a91818d9978 P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 2c4d718738d1408eabce489f4e140af6, 9808124b81e14d1290ab0a6641a4b836; no voters:
I20240501 08:13:35.634096 6406 raft_consensus.cc:2781] T 97943d9ec11f4c5da7a33a91818d9978 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:35.634269 6406 raft_consensus.cc:686] T 97943d9ec11f4c5da7a33a91818d9978 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Becoming Leader. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Running, Role: LEADER
I20240501 08:13:35.634546 6406 consensus_queue.cc:237] T 97943d9ec11f4c5da7a33a91818d9978 P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:35.637338 4903 catalog_manager.cc:5484] T 97943d9ec11f4c5da7a33a91818d9978 P 0a952997a9fc4bfb8d7cbc56de6e0598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130). New cstate: current_term: 1 leader_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:35.675726 6262 raft_consensus.cc:482] T e91cc538a8154588ac05a1e7e508b472 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:35.675940 6262 raft_consensus.cc:504] T e91cc538a8154588ac05a1e7e508b472 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:35.676762 6262 leader_election.cc:288] T e91cc538a8154588ac05a1e7e508b472 P b6cc443a494e4595b0e7cf1777620a3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937), 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983)
I20240501 08:13:35.677405 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e91cc538a8154588ac05a1e7e508b472" candidate_uuid: "b6cc443a494e4595b0e7cf1777620a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808124b81e14d1290ab0a6641a4b836" is_pre_election: true
I20240501 08:13:35.677690 5311 raft_consensus.cc:2454] T e91cc538a8154588ac05a1e7e508b472 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6cc443a494e4595b0e7cf1777620a3f in term 0.
I20240501 08:13:35.678128 4986 leader_election.cc:302] T e91cc538a8154588ac05a1e7e508b472 P b6cc443a494e4595b0e7cf1777620a3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9808124b81e14d1290ab0a6641a4b836, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:35.678443 6262 raft_consensus.cc:2781] T e91cc538a8154588ac05a1e7e508b472 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:35.678598 6262 raft_consensus.cc:482] T e91cc538a8154588ac05a1e7e508b472 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:35.678737 6262 raft_consensus.cc:3037] T e91cc538a8154588ac05a1e7e508b472 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:35.681174 6262 raft_consensus.cc:504] T e91cc538a8154588ac05a1e7e508b472 P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:35.682076 6262 leader_election.cc:288] T e91cc538a8154588ac05a1e7e508b472 P b6cc443a494e4595b0e7cf1777620a3f [CANDIDATE]: Term 1 election: Requested vote from peers 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937), 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983)
I20240501 08:13:35.682330 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e91cc538a8154588ac05a1e7e508b472" candidate_uuid: "b6cc443a494e4595b0e7cf1777620a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
I20240501 08:13:35.682535 5311 raft_consensus.cc:3037] T e91cc538a8154588ac05a1e7e508b472 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:35.683882 5180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e91cc538a8154588ac05a1e7e508b472" candidate_uuid: "b6cc443a494e4595b0e7cf1777620a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" is_pre_election: true
I20240501 08:13:35.684108 5180 raft_consensus.cc:2454] T e91cc538a8154588ac05a1e7e508b472 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6cc443a494e4595b0e7cf1777620a3f in term 0.
I20240501 08:13:35.684041 5181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e91cc538a8154588ac05a1e7e508b472" candidate_uuid: "b6cc443a494e4595b0e7cf1777620a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:35.684302 5181 raft_consensus.cc:3037] T e91cc538a8154588ac05a1e7e508b472 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:35.684808 5311 raft_consensus.cc:2454] T e91cc538a8154588ac05a1e7e508b472 P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6cc443a494e4595b0e7cf1777620a3f in term 1.
I20240501 08:13:35.685161 4986 leader_election.cc:302] T e91cc538a8154588ac05a1e7e508b472 P b6cc443a494e4595b0e7cf1777620a3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9808124b81e14d1290ab0a6641a4b836, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:35.685438 6262 raft_consensus.cc:2781] T e91cc538a8154588ac05a1e7e508b472 P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:35.685839 6262 raft_consensus.cc:686] T e91cc538a8154588ac05a1e7e508b472 P b6cc443a494e4595b0e7cf1777620a3f [term 1 LEADER]: Becoming Leader. State: Replica: b6cc443a494e4595b0e7cf1777620a3f, State: Running, Role: LEADER
I20240501 08:13:35.686141 6262 consensus_queue.cc:237] T e91cc538a8154588ac05a1e7e508b472 P b6cc443a494e4595b0e7cf1777620a3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:35.689044 4903 catalog_manager.cc:5484] T e91cc538a8154588ac05a1e7e508b472 P b6cc443a494e4595b0e7cf1777620a3f reported cstate change: term changed from 0 to 1, leader changed from <none> to b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129). New cstate: current_term: 1 leader_uuid: "b6cc443a494e4595b0e7cf1777620a3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:35.690399 6406 raft_consensus.cc:482] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:35.690582 6406 raft_consensus.cc:504] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:35.691306 6406 leader_election.cc:288] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983), 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797)
I20240501 08:13:35.691937 5441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f518dc436d1d4e9dbe0ec9bbaf17e5fd" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4d718738d1408eabce489f4e140af6" is_pre_election: true
I20240501 08:13:35.692203 5441 raft_consensus.cc:2454] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 0.
I20240501 08:13:35.692473 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f518dc436d1d4e9dbe0ec9bbaf17e5fd" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808124b81e14d1290ab0a6641a4b836" is_pre_election: true
I20240501 08:13:35.692658 5311 raft_consensus.cc:2454] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 0.
I20240501 08:13:35.692656 5118 leader_election.cc:302] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 2c4d718738d1408eabce489f4e140af6; no voters:
I20240501 08:13:35.692961 6406 raft_consensus.cc:2781] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:35.693089 6406 raft_consensus.cc:482] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:35.693178 6406 raft_consensus.cc:3037] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:35.704547 5181 raft_consensus.cc:2454] T e91cc538a8154588ac05a1e7e508b472 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6cc443a494e4595b0e7cf1777620a3f in term 1.
I20240501 08:13:35.704499 6406 raft_consensus.cc:504] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:35.705240 6406 leader_election.cc:288] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 election: Requested vote from peers 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983), 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797)
I20240501 08:13:35.705750 5441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f518dc436d1d4e9dbe0ec9bbaf17e5fd" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:35.705766 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f518dc436d1d4e9dbe0ec9bbaf17e5fd" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
I20240501 08:13:35.706003 5311 raft_consensus.cc:3037] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:35.706048 5441 raft_consensus.cc:3037] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:35.708842 5441 raft_consensus.cc:2454] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 1.
I20240501 08:13:35.708856 5311 raft_consensus.cc:2454] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 1.
I20240501 08:13:35.709323 5117 leader_election.cc:302] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 2c4d718738d1408eabce489f4e140af6, 9808124b81e14d1290ab0a6641a4b836; no voters:
I20240501 08:13:35.709697 6406 raft_consensus.cc:2781] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:35.709833 6406 raft_consensus.cc:686] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Becoming Leader. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Running, Role: LEADER
I20240501 08:13:35.710112 6406 consensus_queue.cc:237] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:35.713124 4903 catalog_manager.cc:5484] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 0a952997a9fc4bfb8d7cbc56de6e0598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130). New cstate: current_term: 1 leader_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: HEALTHY } } }
W20240501 08:13:36.081349 6496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:36.081696 6496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:13:36.089224 6496 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240501 08:13:36.137820 6406 consensus_queue.cc:1035] T 97943d9ec11f4c5da7a33a91818d9978 P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:36.143121 6406 consensus_queue.cc:1035] T 97943d9ec11f4c5da7a33a91818d9978 P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:36.168004 6262 consensus_queue.cc:1035] T e91cc538a8154588ac05a1e7e508b472 P b6cc443a494e4595b0e7cf1777620a3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:36.201262 6525 consensus_queue.cc:1035] T e91cc538a8154588ac05a1e7e508b472 P b6cc443a494e4595b0e7cf1777620a3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:36.264463 5312 raft_consensus.cc:1261] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Refusing update from remote peer 0a952997a9fc4bfb8d7cbc56de6e0598: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:36.264873 5441 raft_consensus.cc:1261] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Refusing update from remote peer 0a952997a9fc4bfb8d7cbc56de6e0598: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:36.265525 6406 consensus_queue.cc:1035] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:36.276996 6415 consensus_queue.cc:1035] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:36.324717 6539 mvcc.cc:205] Tried to move back new op lower bound from 7022801781810503680 to 7022801779550932992. Current Snapshot: MvccSnapshot[applied={T|T < 7022801781810503680}]
I20240501 08:13:36.401461 6540 mvcc.cc:205] Tried to move back new op lower bound from 7022801781810503680 to 7022801779550932992. Current Snapshot: MvccSnapshot[applied={T|T < 7022801781810503680}]
INSERT report
rows total: 2000
time total: 268.134 ms
time per row: 0.134067 ms
I20240501 08:13:36.512681 4908 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33308:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:13:36.521742 5161 tablet_service.cc:1467] Processing CreateTablet for tablet 2e0d1fd6767b454e9871e1bdec86caa8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=4ba2cf767bcc483e85c5eb10b0a50429]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:36.522225 5161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e0d1fd6767b454e9871e1bdec86caa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:36.522343 5029 tablet_service.cc:1467] Processing CreateTablet for tablet 2e0d1fd6767b454e9871e1bdec86caa8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=4ba2cf767bcc483e85c5eb10b0a50429]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:36.522733 5029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e0d1fd6767b454e9871e1bdec86caa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:36.526603 5290 tablet_service.cc:1467] Processing CreateTablet for tablet e13f0b2e652c44fc994e9f950d40becb (DEFAULT_TABLE table=rebalance_test_table_6 [id=4ba2cf767bcc483e85c5eb10b0a50429]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:36.526682 5031 tablet_service.cc:1467] Processing CreateTablet for tablet e13f0b2e652c44fc994e9f950d40becb (DEFAULT_TABLE table=rebalance_test_table_6 [id=4ba2cf767bcc483e85c5eb10b0a50429]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:36.527042 5290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e13f0b2e652c44fc994e9f950d40becb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:36.527052 5031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e13f0b2e652c44fc994e9f950d40becb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:36.527308 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 2e0d1fd6767b454e9871e1bdec86caa8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=4ba2cf767bcc483e85c5eb10b0a50429]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:36.527714 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e0d1fd6767b454e9871e1bdec86caa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:36.527896 5420 tablet_service.cc:1467] Processing CreateTablet for tablet e13f0b2e652c44fc994e9f950d40becb (DEFAULT_TABLE table=rebalance_test_table_6 [id=4ba2cf767bcc483e85c5eb10b0a50429]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:36.528210 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e13f0b2e652c44fc994e9f950d40becb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:36.534513 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 1835b9a1f198420c82e1994a3056ea9a (DEFAULT_TABLE table=rebalance_test_table_6 [id=4ba2cf767bcc483e85c5eb10b0a50429]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:36.535048 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1835b9a1f198420c82e1994a3056ea9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:36.538550 5160 tablet_service.cc:1467] Processing CreateTablet for tablet 1835b9a1f198420c82e1994a3056ea9a (DEFAULT_TABLE table=rebalance_test_table_6 [id=4ba2cf767bcc483e85c5eb10b0a50429]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:36.539163 5160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1835b9a1f198420c82e1994a3056ea9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:36.542382 6564 tablet_bootstrap.cc:492] T e13f0b2e652c44fc994e9f950d40becb P 9808124b81e14d1290ab0a6641a4b836: Bootstrap starting.
I20240501 08:13:36.544296 6564 tablet_bootstrap.cc:654] T e13f0b2e652c44fc994e9f950d40becb P 9808124b81e14d1290ab0a6641a4b836: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:36.544961 5030 tablet_service.cc:1467] Processing CreateTablet for tablet 1835b9a1f198420c82e1994a3056ea9a (DEFAULT_TABLE table=rebalance_test_table_6 [id=4ba2cf767bcc483e85c5eb10b0a50429]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:36.545413 5030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1835b9a1f198420c82e1994a3056ea9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:36.546653 6564 tablet_bootstrap.cc:492] T e13f0b2e652c44fc994e9f950d40becb P 9808124b81e14d1290ab0a6641a4b836: No bootstrap required, opened a new log
I20240501 08:13:36.546788 6564 ts_tablet_manager.cc:1367] T e13f0b2e652c44fc994e9f950d40becb P 9808124b81e14d1290ab0a6641a4b836: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240501 08:13:36.547547 6564 raft_consensus.cc:348] T e13f0b2e652c44fc994e9f950d40becb P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:36.547752 6564 raft_consensus.cc:374] T e13f0b2e652c44fc994e9f950d40becb P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:36.547824 6564 raft_consensus.cc:729] T e13f0b2e652c44fc994e9f950d40becb P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Initialized, Role: FOLLOWER
I20240501 08:13:36.548014 6564 consensus_queue.cc:260] T e13f0b2e652c44fc994e9f950d40becb P 9808124b81e14d1290ab0a6641a4b836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:36.549026 6565 tablet_bootstrap.cc:492] T 2e0d1fd6767b454e9871e1bdec86caa8 P 2c4d718738d1408eabce489f4e140af6: Bootstrap starting.
I20240501 08:13:36.549451 6564 ts_tablet_manager.cc:1398] T e13f0b2e652c44fc994e9f950d40becb P 9808124b81e14d1290ab0a6641a4b836: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:13:36.553243 6565 tablet_bootstrap.cc:654] T 2e0d1fd6767b454e9871e1bdec86caa8 P 2c4d718738d1408eabce489f4e140af6: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:36.563985 6568 tablet_bootstrap.cc:492] T 2e0d1fd6767b454e9871e1bdec86caa8 P b6cc443a494e4595b0e7cf1777620a3f: Bootstrap starting.
I20240501 08:13:36.564280 6563 tablet_bootstrap.cc:492] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598: Bootstrap starting.
I20240501 08:13:36.566155 6563 tablet_bootstrap.cc:654] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:36.566759 6568 tablet_bootstrap.cc:654] T 2e0d1fd6767b454e9871e1bdec86caa8 P b6cc443a494e4595b0e7cf1777620a3f: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:36.569036 6565 tablet_bootstrap.cc:492] T 2e0d1fd6767b454e9871e1bdec86caa8 P 2c4d718738d1408eabce489f4e140af6: No bootstrap required, opened a new log
I20240501 08:13:36.569205 6565 ts_tablet_manager.cc:1367] T 2e0d1fd6767b454e9871e1bdec86caa8 P 2c4d718738d1408eabce489f4e140af6: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.003s
I20240501 08:13:36.569209 6563 tablet_bootstrap.cc:492] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598: No bootstrap required, opened a new log
I20240501 08:13:36.569398 6563 ts_tablet_manager.cc:1367] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240501 08:13:36.570020 6565 raft_consensus.cc:348] T 2e0d1fd6767b454e9871e1bdec86caa8 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:36.570230 6565 raft_consensus.cc:374] T 2e0d1fd6767b454e9871e1bdec86caa8 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:36.570302 6565 raft_consensus.cc:729] T 2e0d1fd6767b454e9871e1bdec86caa8 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Initialized, Role: FOLLOWER
I20240501 08:13:36.570503 6565 consensus_queue.cc:260] T 2e0d1fd6767b454e9871e1bdec86caa8 P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:36.570494 6563 raft_consensus.cc:348] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:36.570789 6563 raft_consensus.cc:374] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:36.570896 6563 raft_consensus.cc:729] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Initialized, Role: FOLLOWER
I20240501 08:13:36.570986 6565 ts_tablet_manager.cc:1398] T 2e0d1fd6767b454e9871e1bdec86caa8 P 2c4d718738d1408eabce489f4e140af6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:36.571257 6565 tablet_bootstrap.cc:492] T e13f0b2e652c44fc994e9f950d40becb P 2c4d718738d1408eabce489f4e140af6: Bootstrap starting.
I20240501 08:13:36.571219 6563 consensus_queue.cc:260] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:36.571854 6563 ts_tablet_manager.cc:1398] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:13:36.572427 6563 tablet_bootstrap.cc:492] T 1835b9a1f198420c82e1994a3056ea9a P 0a952997a9fc4bfb8d7cbc56de6e0598: Bootstrap starting.
I20240501 08:13:36.574026 6563 tablet_bootstrap.cc:654] T 1835b9a1f198420c82e1994a3056ea9a P 0a952997a9fc4bfb8d7cbc56de6e0598: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:36.574492 6568 tablet_bootstrap.cc:492] T 2e0d1fd6767b454e9871e1bdec86caa8 P b6cc443a494e4595b0e7cf1777620a3f: No bootstrap required, opened a new log
I20240501 08:13:36.574764 6568 ts_tablet_manager.cc:1367] T 2e0d1fd6767b454e9871e1bdec86caa8 P b6cc443a494e4595b0e7cf1777620a3f: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20240501 08:13:36.576465 6565 tablet_bootstrap.cc:654] T e13f0b2e652c44fc994e9f950d40becb P 2c4d718738d1408eabce489f4e140af6: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:36.577159 6568 raft_consensus.cc:348] T 2e0d1fd6767b454e9871e1bdec86caa8 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:36.577466 6568 raft_consensus.cc:374] T 2e0d1fd6767b454e9871e1bdec86caa8 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:36.577606 6568 raft_consensus.cc:729] T 2e0d1fd6767b454e9871e1bdec86caa8 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6cc443a494e4595b0e7cf1777620a3f, State: Initialized, Role: FOLLOWER
I20240501 08:13:36.577896 6568 consensus_queue.cc:260] T 2e0d1fd6767b454e9871e1bdec86caa8 P b6cc443a494e4595b0e7cf1777620a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:36.578048 6406 raft_consensus.cc:482] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:36.578336 6406 raft_consensus.cc:504] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:36.579071 6406 leader_election.cc:288] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417), 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797)
I20240501 08:13:36.578912 6568 ts_tablet_manager.cc:1398] T 2e0d1fd6767b454e9871e1bdec86caa8 P b6cc443a494e4595b0e7cf1777620a3f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:13:36.579605 6568 tablet_bootstrap.cc:492] T e13f0b2e652c44fc994e9f950d40becb P b6cc443a494e4595b0e7cf1777620a3f: Bootstrap starting.
I20240501 08:13:36.579736 5049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0d1fd6767b454e9871e1bdec86caa8" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f" is_pre_election: true
I20240501 08:13:36.580054 5049 raft_consensus.cc:2454] T 2e0d1fd6767b454e9871e1bdec86caa8 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 0.
I20240501 08:13:36.580996 5439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0d1fd6767b454e9871e1bdec86caa8" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4d718738d1408eabce489f4e140af6" is_pre_election: true
I20240501 08:13:36.581322 5439 raft_consensus.cc:2454] T 2e0d1fd6767b454e9871e1bdec86caa8 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 0.
I20240501 08:13:36.581858 5117 leader_election.cc:302] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:36.582424 6406 raft_consensus.cc:2781] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:36.582563 6406 raft_consensus.cc:482] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:36.582669 6406 raft_consensus.cc:3037] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:36.583747 6568 tablet_bootstrap.cc:654] T e13f0b2e652c44fc994e9f950d40becb P b6cc443a494e4595b0e7cf1777620a3f: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:36.586570 6406 raft_consensus.cc:504] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:36.588179 5049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0d1fd6767b454e9871e1bdec86caa8" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:36.588495 5049 raft_consensus.cc:3037] T 2e0d1fd6767b454e9871e1bdec86caa8 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:36.592118 5049 raft_consensus.cc:2454] T 2e0d1fd6767b454e9871e1bdec86caa8 P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 1.
I20240501 08:13:36.592711 5117 leader_election.cc:302] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:36.593169 6415 raft_consensus.cc:2781] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:36.593384 6415 raft_consensus.cc:686] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Becoming Leader. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Running, Role: LEADER
I20240501 08:13:36.593780 6415 consensus_queue.cc:237] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:36.597505 5439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0d1fd6767b454e9871e1bdec86caa8" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:36.597818 5439 raft_consensus.cc:3037] T 2e0d1fd6767b454e9871e1bdec86caa8 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:36.597909 6563 tablet_bootstrap.cc:492] T 1835b9a1f198420c82e1994a3056ea9a P 0a952997a9fc4bfb8d7cbc56de6e0598: No bootstrap required, opened a new log
I20240501 08:13:36.598114 6563 ts_tablet_manager.cc:1367] T 1835b9a1f198420c82e1994a3056ea9a P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.005s
I20240501 08:13:36.599023 6563 raft_consensus.cc:348] T 1835b9a1f198420c82e1994a3056ea9a P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:36.599319 6563 raft_consensus.cc:374] T 1835b9a1f198420c82e1994a3056ea9a P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:36.599400 6563 raft_consensus.cc:729] T 1835b9a1f198420c82e1994a3056ea9a P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Initialized, Role: FOLLOWER
I20240501 08:13:36.599678 6563 consensus_queue.cc:260] T 1835b9a1f198420c82e1994a3056ea9a P 0a952997a9fc4bfb8d7cbc56de6e0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:36.600371 6563 ts_tablet_manager.cc:1398] T 1835b9a1f198420c82e1994a3056ea9a P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:13:36.602104 5439 raft_consensus.cc:2454] T 2e0d1fd6767b454e9871e1bdec86caa8 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 1.
I20240501 08:13:36.602747 6406 leader_election.cc:288] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 election: Requested vote from peers b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417), 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797)
I20240501 08:13:36.603374 6565 tablet_bootstrap.cc:492] T e13f0b2e652c44fc994e9f950d40becb P 2c4d718738d1408eabce489f4e140af6: No bootstrap required, opened a new log
I20240501 08:13:36.603554 6565 ts_tablet_manager.cc:1367] T e13f0b2e652c44fc994e9f950d40becb P 2c4d718738d1408eabce489f4e140af6: Time spent bootstrapping tablet: real 0.032s user 0.009s sys 0.014s
I20240501 08:13:36.603730 6568 tablet_bootstrap.cc:492] T e13f0b2e652c44fc994e9f950d40becb P b6cc443a494e4595b0e7cf1777620a3f: No bootstrap required, opened a new log
I20240501 08:13:36.603899 6568 ts_tablet_manager.cc:1367] T e13f0b2e652c44fc994e9f950d40becb P b6cc443a494e4595b0e7cf1777620a3f: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.016s
I20240501 08:13:36.604346 6565 raft_consensus.cc:348] T e13f0b2e652c44fc994e9f950d40becb P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:36.604525 6565 raft_consensus.cc:374] T e13f0b2e652c44fc994e9f950d40becb P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:36.604601 6565 raft_consensus.cc:729] T e13f0b2e652c44fc994e9f950d40becb P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Initialized, Role: FOLLOWER
I20240501 08:13:36.604583 6568 raft_consensus.cc:348] T e13f0b2e652c44fc994e9f950d40becb P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:36.604812 6568 raft_consensus.cc:374] T e13f0b2e652c44fc994e9f950d40becb P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:36.604890 6568 raft_consensus.cc:729] T e13f0b2e652c44fc994e9f950d40becb P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6cc443a494e4595b0e7cf1777620a3f, State: Initialized, Role: FOLLOWER
I20240501 08:13:36.604846 6565 consensus_queue.cc:260] T e13f0b2e652c44fc994e9f950d40becb P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:36.605158 6568 consensus_queue.cc:260] T e13f0b2e652c44fc994e9f950d40becb P b6cc443a494e4595b0e7cf1777620a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:36.605500 6565 ts_tablet_manager.cc:1398] T e13f0b2e652c44fc994e9f950d40becb P 2c4d718738d1408eabce489f4e140af6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:13:36.605254 4908 catalog_manager.cc:5484] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130). New cstate: current_term: 1 leader_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:36.605762 6565 tablet_bootstrap.cc:492] T 1835b9a1f198420c82e1994a3056ea9a P 2c4d718738d1408eabce489f4e140af6: Bootstrap starting.
I20240501 08:13:36.607442 6568 ts_tablet_manager.cc:1398] T e13f0b2e652c44fc994e9f950d40becb P b6cc443a494e4595b0e7cf1777620a3f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:13:36.607816 6568 tablet_bootstrap.cc:492] T 1835b9a1f198420c82e1994a3056ea9a P b6cc443a494e4595b0e7cf1777620a3f: Bootstrap starting.
I20240501 08:13:36.609436 6565 tablet_bootstrap.cc:654] T 1835b9a1f198420c82e1994a3056ea9a P 2c4d718738d1408eabce489f4e140af6: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:36.611564 6568 tablet_bootstrap.cc:654] T 1835b9a1f198420c82e1994a3056ea9a P b6cc443a494e4595b0e7cf1777620a3f: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:36.612147 6565 tablet_bootstrap.cc:492] T 1835b9a1f198420c82e1994a3056ea9a P 2c4d718738d1408eabce489f4e140af6: No bootstrap required, opened a new log
I20240501 08:13:36.612324 6565 ts_tablet_manager.cc:1367] T 1835b9a1f198420c82e1994a3056ea9a P 2c4d718738d1408eabce489f4e140af6: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240501 08:13:36.613149 6565 raft_consensus.cc:348] T 1835b9a1f198420c82e1994a3056ea9a P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:36.613328 6565 raft_consensus.cc:374] T 1835b9a1f198420c82e1994a3056ea9a P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:36.613410 6565 raft_consensus.cc:729] T 1835b9a1f198420c82e1994a3056ea9a P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Initialized, Role: FOLLOWER
I20240501 08:13:36.613459 6568 tablet_bootstrap.cc:492] T 1835b9a1f198420c82e1994a3056ea9a P b6cc443a494e4595b0e7cf1777620a3f: No bootstrap required, opened a new log
I20240501 08:13:36.613613 6568 ts_tablet_manager.cc:1367] T 1835b9a1f198420c82e1994a3056ea9a P b6cc443a494e4595b0e7cf1777620a3f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240501 08:13:36.613613 6565 consensus_queue.cc:260] T 1835b9a1f198420c82e1994a3056ea9a P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:36.614204 6565 ts_tablet_manager.cc:1398] T 1835b9a1f198420c82e1994a3056ea9a P 2c4d718738d1408eabce489f4e140af6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:36.614421 6568 raft_consensus.cc:348] T 1835b9a1f198420c82e1994a3056ea9a P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:36.614645 6568 raft_consensus.cc:374] T 1835b9a1f198420c82e1994a3056ea9a P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:36.614742 6568 raft_consensus.cc:729] T 1835b9a1f198420c82e1994a3056ea9a P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6cc443a494e4595b0e7cf1777620a3f, State: Initialized, Role: FOLLOWER
I20240501 08:13:36.614949 6568 consensus_queue.cc:260] T 1835b9a1f198420c82e1994a3056ea9a P b6cc443a494e4595b0e7cf1777620a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:36.615464 6568 ts_tablet_manager.cc:1398] T 1835b9a1f198420c82e1994a3056ea9a P b6cc443a494e4595b0e7cf1777620a3f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:13:36.697397 6445 raft_consensus.cc:482] T e13f0b2e652c44fc994e9f950d40becb P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:36.697613 6445 raft_consensus.cc:504] T e13f0b2e652c44fc994e9f950d40becb P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:36.698288 6445 leader_election.cc:288] T e13f0b2e652c44fc994e9f950d40becb P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983), b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417)
I20240501 08:13:36.698830 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13f0b2e652c44fc994e9f950d40becb" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808124b81e14d1290ab0a6641a4b836" is_pre_election: true
I20240501 08:13:36.698887 5049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13f0b2e652c44fc994e9f950d40becb" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f" is_pre_election: true
I20240501 08:13:36.699062 5311 raft_consensus.cc:2454] T e13f0b2e652c44fc994e9f950d40becb P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 0.
I20240501 08:13:36.699112 5049 raft_consensus.cc:2454] T e13f0b2e652c44fc994e9f950d40becb P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 0.
I20240501 08:13:36.699528 5377 leader_election.cc:302] T e13f0b2e652c44fc994e9f950d40becb P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4d718738d1408eabce489f4e140af6, 9808124b81e14d1290ab0a6641a4b836; no voters:
I20240501 08:13:36.699815 6445 raft_consensus.cc:2781] T e13f0b2e652c44fc994e9f950d40becb P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:36.699937 6445 raft_consensus.cc:482] T e13f0b2e652c44fc994e9f950d40becb P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:36.700026 6445 raft_consensus.cc:3037] T e13f0b2e652c44fc994e9f950d40becb P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:36.701959 6445 raft_consensus.cc:504] T e13f0b2e652c44fc994e9f950d40becb P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:36.702466 6445 leader_election.cc:288] T e13f0b2e652c44fc994e9f950d40becb P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 1 election: Requested vote from peers 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983), b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417)
I20240501 08:13:36.702972 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13f0b2e652c44fc994e9f950d40becb" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
I20240501 08:13:36.703032 5049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13f0b2e652c44fc994e9f950d40becb" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:36.703172 5311 raft_consensus.cc:3037] T e13f0b2e652c44fc994e9f950d40becb P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:36.703233 5049 raft_consensus.cc:3037] T e13f0b2e652c44fc994e9f950d40becb P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:36.705034 5311 raft_consensus.cc:2454] T e13f0b2e652c44fc994e9f950d40becb P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 1.
I20240501 08:13:36.705039 5049 raft_consensus.cc:2454] T e13f0b2e652c44fc994e9f950d40becb P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 1.
I20240501 08:13:36.705453 5377 leader_election.cc:302] T e13f0b2e652c44fc994e9f950d40becb P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4d718738d1408eabce489f4e140af6, 9808124b81e14d1290ab0a6641a4b836; no voters:
I20240501 08:13:36.705729 6445 raft_consensus.cc:2781] T e13f0b2e652c44fc994e9f950d40becb P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:36.706153 6445 raft_consensus.cc:686] T e13f0b2e652c44fc994e9f950d40becb P 2c4d718738d1408eabce489f4e140af6 [term 1 LEADER]: Becoming Leader. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:36.706457 6445 consensus_queue.cc:237] T e13f0b2e652c44fc994e9f950d40becb P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:36.709398 4908 catalog_manager.cc:5484] T e13f0b2e652c44fc994e9f950d40becb P 2c4d718738d1408eabce489f4e140af6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c4d718738d1408eabce489f4e140af6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "2c4d718738d1408eabce489f4e140af6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: HEALTHY } } }
I20240501 08:13:36.896262 6554 raft_consensus.cc:482] T 1835b9a1f198420c82e1994a3056ea9a P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:36.896452 6554 raft_consensus.cc:504] T 1835b9a1f198420c82e1994a3056ea9a P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:36.897110 6554 leader_election.cc:288] T 1835b9a1f198420c82e1994a3056ea9a P b6cc443a494e4595b0e7cf1777620a3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937), 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797)
I20240501 08:13:36.897581 5181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1835b9a1f198420c82e1994a3056ea9a" candidate_uuid: "b6cc443a494e4595b0e7cf1777620a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" is_pre_election: true
I20240501 08:13:36.897593 5439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1835b9a1f198420c82e1994a3056ea9a" candidate_uuid: "b6cc443a494e4595b0e7cf1777620a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4d718738d1408eabce489f4e140af6" is_pre_election: true
I20240501 08:13:36.897842 5181 raft_consensus.cc:2454] T 1835b9a1f198420c82e1994a3056ea9a P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6cc443a494e4595b0e7cf1777620a3f in term 0.
I20240501 08:13:36.897867 5439 raft_consensus.cc:2454] T 1835b9a1f198420c82e1994a3056ea9a P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6cc443a494e4595b0e7cf1777620a3f in term 0.
I20240501 08:13:36.898324 4985 leader_election.cc:302] T 1835b9a1f198420c82e1994a3056ea9a P b6cc443a494e4595b0e7cf1777620a3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:36.898624 6554 raft_consensus.cc:2781] T 1835b9a1f198420c82e1994a3056ea9a P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:36.898734 6554 raft_consensus.cc:482] T 1835b9a1f198420c82e1994a3056ea9a P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:36.898824 6554 raft_consensus.cc:3037] T 1835b9a1f198420c82e1994a3056ea9a P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:36.901225 6554 raft_consensus.cc:504] T 1835b9a1f198420c82e1994a3056ea9a P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:36.901752 6554 leader_election.cc:288] T 1835b9a1f198420c82e1994a3056ea9a P b6cc443a494e4595b0e7cf1777620a3f [CANDIDATE]: Term 1 election: Requested vote from peers 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937), 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797)
I20240501 08:13:36.902158 5181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1835b9a1f198420c82e1994a3056ea9a" candidate_uuid: "b6cc443a494e4595b0e7cf1777620a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:36.902257 5439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1835b9a1f198420c82e1994a3056ea9a" candidate_uuid: "b6cc443a494e4595b0e7cf1777620a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:36.902352 5181 raft_consensus.cc:3037] T 1835b9a1f198420c82e1994a3056ea9a P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:36.902485 5439 raft_consensus.cc:3037] T 1835b9a1f198420c82e1994a3056ea9a P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:36.904623 5181 raft_consensus.cc:2454] T 1835b9a1f198420c82e1994a3056ea9a P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6cc443a494e4595b0e7cf1777620a3f in term 1.
I20240501 08:13:36.904772 5439 raft_consensus.cc:2454] T 1835b9a1f198420c82e1994a3056ea9a P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6cc443a494e4595b0e7cf1777620a3f in term 1.
I20240501 08:13:36.905099 4985 leader_election.cc:302] T 1835b9a1f198420c82e1994a3056ea9a P b6cc443a494e4595b0e7cf1777620a3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:36.905405 6554 raft_consensus.cc:2781] T 1835b9a1f198420c82e1994a3056ea9a P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:36.905846 6554 raft_consensus.cc:686] T 1835b9a1f198420c82e1994a3056ea9a P b6cc443a494e4595b0e7cf1777620a3f [term 1 LEADER]: Becoming Leader. State: Replica: b6cc443a494e4595b0e7cf1777620a3f, State: Running, Role: LEADER
I20240501 08:13:36.906240 6554 consensus_queue.cc:237] T 1835b9a1f198420c82e1994a3056ea9a P b6cc443a494e4595b0e7cf1777620a3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:36.918551 4908 catalog_manager.cc:5484] T 1835b9a1f198420c82e1994a3056ea9a P b6cc443a494e4595b0e7cf1777620a3f reported cstate change: term changed from 0 to 1, leader changed from <none> to b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129). New cstate: current_term: 1 leader_uuid: "b6cc443a494e4595b0e7cf1777620a3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: HEALTHY } } }
I20240501 08:13:37.061018 6406 consensus_queue.cc:1035] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:37.069417 6549 consensus_queue.cc:1035] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:37.095870 6445 consensus_queue.cc:1035] T e13f0b2e652c44fc994e9f950d40becb P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:37.101286 6577 consensus_queue.cc:1035] T e13f0b2e652c44fc994e9f950d40becb P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:13:37.340312 6581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:37.340678 6581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:13:37.348590 6581 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240501 08:13:37.412642 6554 consensus_queue.cc:1035] T 1835b9a1f198420c82e1994a3056ea9a P b6cc443a494e4595b0e7cf1777620a3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:37.421802 6614 consensus_queue.cc:1035] T 1835b9a1f198420c82e1994a3056ea9a P b6cc443a494e4595b0e7cf1777620a3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 230.071 ms
time per row: 0.115035 ms
I20240501 08:13:37.740007 4908 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33308:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:13:37.749380 5290 tablet_service.cc:1467] Processing CreateTablet for tablet 5169fa71adfc4a9d8c9feae475f2b14b (DEFAULT_TABLE table=rebalance_test_table_7 [id=39bc86fa9c434675a1b394e5a958806e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:37.749616 5160 tablet_service.cc:1467] Processing CreateTablet for tablet 5169fa71adfc4a9d8c9feae475f2b14b (DEFAULT_TABLE table=rebalance_test_table_7 [id=39bc86fa9c434675a1b394e5a958806e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:37.749900 5290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5169fa71adfc4a9d8c9feae475f2b14b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:37.750104 5160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5169fa71adfc4a9d8c9feae475f2b14b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:37.750314 5029 tablet_service.cc:1467] Processing CreateTablet for tablet 5169fa71adfc4a9d8c9feae475f2b14b (DEFAULT_TABLE table=rebalance_test_table_7 [id=39bc86fa9c434675a1b394e5a958806e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:37.750727 5029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5169fa71adfc4a9d8c9feae475f2b14b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:37.754720 5030 tablet_service.cc:1467] Processing CreateTablet for tablet 19ec6a8b857d428d858fa831e825c578 (DEFAULT_TABLE table=rebalance_test_table_7 [id=39bc86fa9c434675a1b394e5a958806e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:37.755002 5421 tablet_service.cc:1467] Processing CreateTablet for tablet ab68600377eb4581a724b714cfe234d7 (DEFAULT_TABLE table=rebalance_test_table_7 [id=39bc86fa9c434675a1b394e5a958806e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:37.755151 5030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19ec6a8b857d428d858fa831e825c578. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:37.755370 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab68600377eb4581a724b714cfe234d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:37.760582 6644 tablet_bootstrap.cc:492] T ab68600377eb4581a724b714cfe234d7 P 2c4d718738d1408eabce489f4e140af6: Bootstrap starting.
I20240501 08:13:37.760540 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 19ec6a8b857d428d858fa831e825c578 (DEFAULT_TABLE table=rebalance_test_table_7 [id=39bc86fa9c434675a1b394e5a958806e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:37.761042 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19ec6a8b857d428d858fa831e825c578. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:37.762684 6644 tablet_bootstrap.cc:654] T ab68600377eb4581a724b714cfe234d7 P 2c4d718738d1408eabce489f4e140af6: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:37.763667 5161 tablet_service.cc:1467] Processing CreateTablet for tablet ab68600377eb4581a724b714cfe234d7 (DEFAULT_TABLE table=rebalance_test_table_7 [id=39bc86fa9c434675a1b394e5a958806e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:37.764241 5161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab68600377eb4581a724b714cfe234d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:37.771044 6644 tablet_bootstrap.cc:492] T ab68600377eb4581a724b714cfe234d7 P 2c4d718738d1408eabce489f4e140af6: No bootstrap required, opened a new log
I20240501 08:13:37.771319 6644 ts_tablet_manager.cc:1367] T ab68600377eb4581a724b714cfe234d7 P 2c4d718738d1408eabce489f4e140af6: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240501 08:13:37.772205 6644 raft_consensus.cc:348] T ab68600377eb4581a724b714cfe234d7 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:37.772505 6644 raft_consensus.cc:374] T ab68600377eb4581a724b714cfe234d7 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:37.772634 6644 raft_consensus.cc:729] T ab68600377eb4581a724b714cfe234d7 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Initialized, Role: FOLLOWER
I20240501 08:13:37.772938 6644 consensus_queue.cc:260] T ab68600377eb4581a724b714cfe234d7 P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:37.774273 6646 tablet_bootstrap.cc:492] T 5169fa71adfc4a9d8c9feae475f2b14b P b6cc443a494e4595b0e7cf1777620a3f: Bootstrap starting.
I20240501 08:13:37.777197 6646 tablet_bootstrap.cc:654] T 5169fa71adfc4a9d8c9feae475f2b14b P b6cc443a494e4595b0e7cf1777620a3f: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:37.778822 6644 ts_tablet_manager.cc:1398] T ab68600377eb4581a724b714cfe234d7 P 2c4d718738d1408eabce489f4e140af6: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240501 08:13:37.779151 6643 tablet_bootstrap.cc:492] T 5169fa71adfc4a9d8c9feae475f2b14b P 0a952997a9fc4bfb8d7cbc56de6e0598: Bootstrap starting.
I20240501 08:13:37.780961 6643 tablet_bootstrap.cc:654] T 5169fa71adfc4a9d8c9feae475f2b14b P 0a952997a9fc4bfb8d7cbc56de6e0598: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:37.783011 6643 tablet_bootstrap.cc:492] T 5169fa71adfc4a9d8c9feae475f2b14b P 0a952997a9fc4bfb8d7cbc56de6e0598: No bootstrap required, opened a new log
I20240501 08:13:37.783108 6646 tablet_bootstrap.cc:492] T 5169fa71adfc4a9d8c9feae475f2b14b P b6cc443a494e4595b0e7cf1777620a3f: No bootstrap required, opened a new log
I20240501 08:13:37.783165 6643 ts_tablet_manager.cc:1367] T 5169fa71adfc4a9d8c9feae475f2b14b P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:13:37.783468 6646 ts_tablet_manager.cc:1367] T 5169fa71adfc4a9d8c9feae475f2b14b P b6cc443a494e4595b0e7cf1777620a3f: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240501 08:13:37.784206 6643 raft_consensus.cc:348] T 5169fa71adfc4a9d8c9feae475f2b14b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:37.784466 6643 raft_consensus.cc:374] T 5169fa71adfc4a9d8c9feae475f2b14b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:37.784372 6646 raft_consensus.cc:348] T 5169fa71adfc4a9d8c9feae475f2b14b P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:37.784569 6646 raft_consensus.cc:374] T 5169fa71adfc4a9d8c9feae475f2b14b P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:37.784595 6643 raft_consensus.cc:729] T 5169fa71adfc4a9d8c9feae475f2b14b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Initialized, Role: FOLLOWER
I20240501 08:13:37.784754 6646 raft_consensus.cc:729] T 5169fa71adfc4a9d8c9feae475f2b14b P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6cc443a494e4595b0e7cf1777620a3f, State: Initialized, Role: FOLLOWER
I20240501 08:13:37.784933 6644 tablet_bootstrap.cc:492] T 19ec6a8b857d428d858fa831e825c578 P 2c4d718738d1408eabce489f4e140af6: Bootstrap starting.
I20240501 08:13:37.784813 6643 consensus_queue.cc:260] T 5169fa71adfc4a9d8c9feae475f2b14b P 0a952997a9fc4bfb8d7cbc56de6e0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:37.785463 6643 ts_tablet_manager.cc:1398] T 5169fa71adfc4a9d8c9feae475f2b14b P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:37.785918 6643 tablet_bootstrap.cc:492] T ab68600377eb4581a724b714cfe234d7 P 0a952997a9fc4bfb8d7cbc56de6e0598: Bootstrap starting.
I20240501 08:13:37.786388 6646 consensus_queue.cc:260] T 5169fa71adfc4a9d8c9feae475f2b14b P b6cc443a494e4595b0e7cf1777620a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:37.787071 6646 ts_tablet_manager.cc:1398] T 5169fa71adfc4a9d8c9feae475f2b14b P b6cc443a494e4595b0e7cf1777620a3f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:13:37.787539 6643 tablet_bootstrap.cc:654] T ab68600377eb4581a724b714cfe234d7 P 0a952997a9fc4bfb8d7cbc56de6e0598: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:37.787585 6646 tablet_bootstrap.cc:492] T 19ec6a8b857d428d858fa831e825c578 P b6cc443a494e4595b0e7cf1777620a3f: Bootstrap starting.
I20240501 08:13:37.788192 6644 tablet_bootstrap.cc:654] T 19ec6a8b857d428d858fa831e825c578 P 2c4d718738d1408eabce489f4e140af6: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:37.792966 6649 tablet_bootstrap.cc:492] T 5169fa71adfc4a9d8c9feae475f2b14b P 9808124b81e14d1290ab0a6641a4b836: Bootstrap starting.
I20240501 08:13:37.793133 5292 tablet_service.cc:1467] Processing CreateTablet for tablet 19ec6a8b857d428d858fa831e825c578 (DEFAULT_TABLE table=rebalance_test_table_7 [id=39bc86fa9c434675a1b394e5a958806e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:37.793594 5292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19ec6a8b857d428d858fa831e825c578. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:37.794854 6646 tablet_bootstrap.cc:654] T 19ec6a8b857d428d858fa831e825c578 P b6cc443a494e4595b0e7cf1777620a3f: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:37.795373 6649 tablet_bootstrap.cc:654] T 5169fa71adfc4a9d8c9feae475f2b14b P 9808124b81e14d1290ab0a6641a4b836: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:37.797991 6614 raft_consensus.cc:482] T 5169fa71adfc4a9d8c9feae475f2b14b P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:37.798238 6614 raft_consensus.cc:504] T 5169fa71adfc4a9d8c9feae475f2b14b P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:37.799250 6614 leader_election.cc:288] T 5169fa71adfc4a9d8c9feae475f2b14b P b6cc443a494e4595b0e7cf1777620a3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983), 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937)
I20240501 08:13:37.800552 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5169fa71adfc4a9d8c9feae475f2b14b" candidate_uuid: "b6cc443a494e4595b0e7cf1777620a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808124b81e14d1290ab0a6641a4b836" is_pre_election: true
I20240501 08:13:37.801633 6649 tablet_bootstrap.cc:492] T 5169fa71adfc4a9d8c9feae475f2b14b P 9808124b81e14d1290ab0a6641a4b836: No bootstrap required, opened a new log
W20240501 08:13:37.801597 4986 leader_election.cc:341] T 5169fa71adfc4a9d8c9feae475f2b14b P b6cc443a494e4595b0e7cf1777620a3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:13:37.802453 6649 ts_tablet_manager.cc:1367] T 5169fa71adfc4a9d8c9feae475f2b14b P 9808124b81e14d1290ab0a6641a4b836: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240501 08:13:37.803879 6649 raft_consensus.cc:348] T 5169fa71adfc4a9d8c9feae475f2b14b P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:37.805330 6649 raft_consensus.cc:374] T 5169fa71adfc4a9d8c9feae475f2b14b P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:37.805409 5180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5169fa71adfc4a9d8c9feae475f2b14b" candidate_uuid: "b6cc443a494e4595b0e7cf1777620a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" is_pre_election: true
I20240501 08:13:37.805640 5180 raft_consensus.cc:2454] T 5169fa71adfc4a9d8c9feae475f2b14b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6cc443a494e4595b0e7cf1777620a3f in term 0.
I20240501 08:13:37.806111 6646 tablet_bootstrap.cc:492] T 19ec6a8b857d428d858fa831e825c578 P b6cc443a494e4595b0e7cf1777620a3f: No bootstrap required, opened a new log
I20240501 08:13:37.806346 6646 ts_tablet_manager.cc:1367] T 19ec6a8b857d428d858fa831e825c578 P b6cc443a494e4595b0e7cf1777620a3f: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.003s
I20240501 08:13:37.807183 6646 raft_consensus.cc:348] T 19ec6a8b857d428d858fa831e825c578 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:37.807499 6646 raft_consensus.cc:374] T 19ec6a8b857d428d858fa831e825c578 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:37.807626 6646 raft_consensus.cc:729] T 19ec6a8b857d428d858fa831e825c578 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6cc443a494e4595b0e7cf1777620a3f, State: Initialized, Role: FOLLOWER
I20240501 08:13:37.807868 4985 leader_election.cc:302] T 5169fa71adfc4a9d8c9feae475f2b14b P b6cc443a494e4595b0e7cf1777620a3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, b6cc443a494e4595b0e7cf1777620a3f; no voters: 9808124b81e14d1290ab0a6641a4b836
I20240501 08:13:37.807946 6646 consensus_queue.cc:260] T 19ec6a8b857d428d858fa831e825c578 P b6cc443a494e4595b0e7cf1777620a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:37.808444 6631 raft_consensus.cc:2781] T 5169fa71adfc4a9d8c9feae475f2b14b P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:37.808534 6631 raft_consensus.cc:482] T 5169fa71adfc4a9d8c9feae475f2b14b P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:37.808640 6631 raft_consensus.cc:3037] T 5169fa71adfc4a9d8c9feae475f2b14b P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:37.811695 6631 raft_consensus.cc:504] T 5169fa71adfc4a9d8c9feae475f2b14b P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:37.812743 6631 leader_election.cc:288] T 5169fa71adfc4a9d8c9feae475f2b14b P b6cc443a494e4595b0e7cf1777620a3f [CANDIDATE]: Term 1 election: Requested vote from peers 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983), 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937)
I20240501 08:13:37.812683 6646 ts_tablet_manager.cc:1398] T 19ec6a8b857d428d858fa831e825c578 P b6cc443a494e4595b0e7cf1777620a3f: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:13:37.814013 5180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5169fa71adfc4a9d8c9feae475f2b14b" candidate_uuid: "b6cc443a494e4595b0e7cf1777620a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:37.814262 5180 raft_consensus.cc:3037] T 5169fa71adfc4a9d8c9feae475f2b14b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:37.813267 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5169fa71adfc4a9d8c9feae475f2b14b" candidate_uuid: "b6cc443a494e4595b0e7cf1777620a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
I20240501 08:13:37.816365 6649 raft_consensus.cc:729] T 5169fa71adfc4a9d8c9feae475f2b14b P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Initialized, Role: FOLLOWER
I20240501 08:13:37.816709 6649 consensus_queue.cc:260] T 5169fa71adfc4a9d8c9feae475f2b14b P 9808124b81e14d1290ab0a6641a4b836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:37.817077 5311 raft_consensus.cc:3037] T 5169fa71adfc4a9d8c9feae475f2b14b P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:37.817183 5180 raft_consensus.cc:2454] T 5169fa71adfc4a9d8c9feae475f2b14b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6cc443a494e4595b0e7cf1777620a3f in term 1.
I20240501 08:13:37.817723 4985 leader_election.cc:302] T 5169fa71adfc4a9d8c9feae475f2b14b P b6cc443a494e4595b0e7cf1777620a3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:37.818253 6614 raft_consensus.cc:2781] T 5169fa71adfc4a9d8c9feae475f2b14b P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:37.818382 6614 raft_consensus.cc:686] T 5169fa71adfc4a9d8c9feae475f2b14b P b6cc443a494e4595b0e7cf1777620a3f [term 1 LEADER]: Becoming Leader. State: Replica: b6cc443a494e4595b0e7cf1777620a3f, State: Running, Role: LEADER
I20240501 08:13:37.818679 6614 consensus_queue.cc:237] T 5169fa71adfc4a9d8c9feae475f2b14b P b6cc443a494e4595b0e7cf1777620a3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:37.819859 5311 raft_consensus.cc:2454] T 5169fa71adfc4a9d8c9feae475f2b14b P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6cc443a494e4595b0e7cf1777620a3f in term 1.
I20240501 08:13:37.820806 6649 ts_tablet_manager.cc:1398] T 5169fa71adfc4a9d8c9feae475f2b14b P 9808124b81e14d1290ab0a6641a4b836: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240501 08:13:37.821888 5291 tablet_service.cc:1467] Processing CreateTablet for tablet ab68600377eb4581a724b714cfe234d7 (DEFAULT_TABLE table=rebalance_test_table_7 [id=39bc86fa9c434675a1b394e5a958806e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:37.822389 5291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab68600377eb4581a724b714cfe234d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:37.833189 4904 catalog_manager.cc:5484] T 5169fa71adfc4a9d8c9feae475f2b14b P b6cc443a494e4595b0e7cf1777620a3f reported cstate change: term changed from 0 to 1, leader changed from <none> to b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129). New cstate: current_term: 1 leader_uuid: "b6cc443a494e4595b0e7cf1777620a3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:37.835173 6649 tablet_bootstrap.cc:492] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836: Bootstrap starting.
I20240501 08:13:37.837318 6643 tablet_bootstrap.cc:492] T ab68600377eb4581a724b714cfe234d7 P 0a952997a9fc4bfb8d7cbc56de6e0598: No bootstrap required, opened a new log
I20240501 08:13:37.837498 6643 ts_tablet_manager.cc:1367] T ab68600377eb4581a724b714cfe234d7 P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent bootstrapping tablet: real 0.052s user 0.014s sys 0.035s
I20240501 08:13:37.837489 6644 tablet_bootstrap.cc:492] T 19ec6a8b857d428d858fa831e825c578 P 2c4d718738d1408eabce489f4e140af6: No bootstrap required, opened a new log
I20240501 08:13:37.837658 6644 ts_tablet_manager.cc:1367] T 19ec6a8b857d428d858fa831e825c578 P 2c4d718738d1408eabce489f4e140af6: Time spent bootstrapping tablet: real 0.053s user 0.015s sys 0.019s
I20240501 08:13:37.838199 6643 raft_consensus.cc:348] T ab68600377eb4581a724b714cfe234d7 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:37.838404 6643 raft_consensus.cc:374] T ab68600377eb4581a724b714cfe234d7 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:37.838475 6643 raft_consensus.cc:729] T ab68600377eb4581a724b714cfe234d7 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Initialized, Role: FOLLOWER
I20240501 08:13:37.838630 6649 tablet_bootstrap.cc:654] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:37.838598 6644 raft_consensus.cc:348] T 19ec6a8b857d428d858fa831e825c578 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:37.838816 6644 raft_consensus.cc:374] T 19ec6a8b857d428d858fa831e825c578 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:37.838723 6643 consensus_queue.cc:260] T ab68600377eb4581a724b714cfe234d7 P 0a952997a9fc4bfb8d7cbc56de6e0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:37.838905 6644 raft_consensus.cc:729] T 19ec6a8b857d428d858fa831e825c578 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Initialized, Role: FOLLOWER
I20240501 08:13:37.839152 6644 consensus_queue.cc:260] T 19ec6a8b857d428d858fa831e825c578 P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:37.839313 6643 ts_tablet_manager.cc:1398] T ab68600377eb4581a724b714cfe234d7 P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:13:37.839711 6644 ts_tablet_manager.cc:1398] T 19ec6a8b857d428d858fa831e825c578 P 2c4d718738d1408eabce489f4e140af6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:13:37.841976 6649 tablet_bootstrap.cc:492] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836: No bootstrap required, opened a new log
I20240501 08:13:37.842156 6649 ts_tablet_manager.cc:1367] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20240501 08:13:37.844404 6649 raft_consensus.cc:348] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:37.844640 6649 raft_consensus.cc:374] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:37.844720 6649 raft_consensus.cc:729] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Initialized, Role: FOLLOWER
I20240501 08:13:37.844954 6649 consensus_queue.cc:260] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:37.845481 6649 ts_tablet_manager.cc:1398] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:13:37.845898 6649 tablet_bootstrap.cc:492] T ab68600377eb4581a724b714cfe234d7 P 9808124b81e14d1290ab0a6641a4b836: Bootstrap starting.
I20240501 08:13:37.847225 6649 tablet_bootstrap.cc:654] T ab68600377eb4581a724b714cfe234d7 P 9808124b81e14d1290ab0a6641a4b836: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:37.849085 6649 tablet_bootstrap.cc:492] T ab68600377eb4581a724b714cfe234d7 P 9808124b81e14d1290ab0a6641a4b836: No bootstrap required, opened a new log
I20240501 08:13:37.849211 6649 ts_tablet_manager.cc:1367] T ab68600377eb4581a724b714cfe234d7 P 9808124b81e14d1290ab0a6641a4b836: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:13:37.849978 6649 raft_consensus.cc:348] T ab68600377eb4581a724b714cfe234d7 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:37.850180 6649 raft_consensus.cc:374] T ab68600377eb4581a724b714cfe234d7 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:37.850276 6649 raft_consensus.cc:729] T ab68600377eb4581a724b714cfe234d7 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Initialized, Role: FOLLOWER
I20240501 08:13:37.850481 6649 consensus_queue.cc:260] T ab68600377eb4581a724b714cfe234d7 P 9808124b81e14d1290ab0a6641a4b836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:37.850958 6649 ts_tablet_manager.cc:1398] T ab68600377eb4581a724b714cfe234d7 P 9808124b81e14d1290ab0a6641a4b836: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:13:38.056016 6445 raft_consensus.cc:482] T ab68600377eb4581a724b714cfe234d7 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:38.056219 6445 raft_consensus.cc:504] T ab68600377eb4581a724b714cfe234d7 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:38.056803 6445 leader_election.cc:288] T ab68600377eb4581a724b714cfe234d7 P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983), 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937)
I20240501 08:13:38.057296 5180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab68600377eb4581a724b714cfe234d7" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" is_pre_election: true
I20240501 08:13:38.057315 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab68600377eb4581a724b714cfe234d7" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808124b81e14d1290ab0a6641a4b836" is_pre_election: true
I20240501 08:13:38.057535 5311 raft_consensus.cc:2454] T ab68600377eb4581a724b714cfe234d7 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 0.
I20240501 08:13:38.057555 5180 raft_consensus.cc:2454] T ab68600377eb4581a724b714cfe234d7 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 0.
I20240501 08:13:38.058024 5376 leader_election.cc:302] T ab68600377eb4581a724b714cfe234d7 P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 2c4d718738d1408eabce489f4e140af6; no voters:
I20240501 08:13:38.058313 6445 raft_consensus.cc:2781] T ab68600377eb4581a724b714cfe234d7 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:38.058440 6445 raft_consensus.cc:482] T ab68600377eb4581a724b714cfe234d7 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:38.058534 6445 raft_consensus.cc:3037] T ab68600377eb4581a724b714cfe234d7 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:38.060353 6445 raft_consensus.cc:504] T ab68600377eb4581a724b714cfe234d7 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:38.060889 6445 leader_election.cc:288] T ab68600377eb4581a724b714cfe234d7 P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 1 election: Requested vote from peers 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983), 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937)
I20240501 08:13:38.061391 5180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab68600377eb4581a724b714cfe234d7" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:38.061390 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab68600377eb4581a724b714cfe234d7" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
I20240501 08:13:38.061566 5311 raft_consensus.cc:3037] T ab68600377eb4581a724b714cfe234d7 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:38.061595 5180 raft_consensus.cc:3037] T ab68600377eb4581a724b714cfe234d7 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:38.063444 5180 raft_consensus.cc:2454] T ab68600377eb4581a724b714cfe234d7 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 1.
I20240501 08:13:38.063611 5311 raft_consensus.cc:2454] T ab68600377eb4581a724b714cfe234d7 P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 1.
I20240501 08:13:38.063864 5376 leader_election.cc:302] T ab68600377eb4581a724b714cfe234d7 P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 2c4d718738d1408eabce489f4e140af6; no voters:
I20240501 08:13:38.064165 6445 raft_consensus.cc:2781] T ab68600377eb4581a724b714cfe234d7 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:38.064289 6445 raft_consensus.cc:686] T ab68600377eb4581a724b714cfe234d7 P 2c4d718738d1408eabce489f4e140af6 [term 1 LEADER]: Becoming Leader. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:38.064525 6445 consensus_queue.cc:237] T ab68600377eb4581a724b714cfe234d7 P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:38.068104 4908 catalog_manager.cc:5484] T ab68600377eb4581a724b714cfe234d7 P 2c4d718738d1408eabce489f4e140af6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c4d718738d1408eabce489f4e140af6 (127.21.84.132). New cstate: current_term: 1 leader_uuid: "2c4d718738d1408eabce489f4e140af6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:38.159379 6647 raft_consensus.cc:482] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:38.159586 6647 raft_consensus.cc:504] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:38.160274 6647 leader_election.cc:288] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417), 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797)
I20240501 08:13:38.160810 5439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ec6a8b857d428d858fa831e825c578" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4d718738d1408eabce489f4e140af6" is_pre_election: true
I20240501 08:13:38.160809 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ec6a8b857d428d858fa831e825c578" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f" is_pre_election: true
I20240501 08:13:38.161052 5439 raft_consensus.cc:2454] T 19ec6a8b857d428d858fa831e825c578 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 0.
I20240501 08:13:38.161052 5051 raft_consensus.cc:2454] T 19ec6a8b857d428d858fa831e825c578 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 0.
I20240501 08:13:38.161571 5248 leader_election.cc:302] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4d718738d1408eabce489f4e140af6, 9808124b81e14d1290ab0a6641a4b836; no voters:
I20240501 08:13:38.161835 6647 raft_consensus.cc:2781] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:38.161972 6647 raft_consensus.cc:482] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:38.162062 6647 raft_consensus.cc:3037] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:38.163906 6647 raft_consensus.cc:504] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:38.164430 6647 leader_election.cc:288] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 1 election: Requested vote from peers b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417), 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797)
I20240501 08:13:38.164870 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ec6a8b857d428d858fa831e825c578" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:38.164891 5439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ec6a8b857d428d858fa831e825c578" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:38.165052 5051 raft_consensus.cc:3037] T 19ec6a8b857d428d858fa831e825c578 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:38.165073 5439 raft_consensus.cc:3037] T 19ec6a8b857d428d858fa831e825c578 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:38.166841 5051 raft_consensus.cc:2454] T 19ec6a8b857d428d858fa831e825c578 P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 1.
I20240501 08:13:38.166839 5439 raft_consensus.cc:2454] T 19ec6a8b857d428d858fa831e825c578 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 1.
I20240501 08:13:38.167225 5248 leader_election.cc:302] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4d718738d1408eabce489f4e140af6, 9808124b81e14d1290ab0a6641a4b836; no voters:
I20240501 08:13:38.167498 6647 raft_consensus.cc:2781] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:38.167639 6647 raft_consensus.cc:686] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836 [term 1 LEADER]: Becoming Leader. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:38.167920 6647 consensus_queue.cc:237] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:38.170821 4908 catalog_manager.cc:5484] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131). New cstate: current_term: 1 leader_uuid: "9808124b81e14d1290ab0a6641a4b836" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:38.234576 6614 consensus_queue.cc:1035] T 5169fa71adfc4a9d8c9feae475f2b14b P b6cc443a494e4595b0e7cf1777620a3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:38.253237 6554 consensus_queue.cc:1035] T 5169fa71adfc4a9d8c9feae475f2b14b P b6cc443a494e4595b0e7cf1777620a3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:13:38.529934 6660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:38.530309 6660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:13:38.537919 6660 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:13:38.574049 6445 consensus_queue.cc:1035] T ab68600377eb4581a724b714cfe234d7 P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:38.582819 6677 consensus_queue.cc:1035] T ab68600377eb4581a724b714cfe234d7 P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_7'
I20240501 08:13:38.612226 6647 consensus_queue.cc:1035] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:38.619076 6528 consensus_queue.cc:1035] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 223.125 ms
time per row: 0.111563 ms
I20240501 08:13:38.944597 4907 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33308:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:13:38.954603 5291 tablet_service.cc:1467] Processing CreateTablet for tablet faf43fec190c4830957d4e3ab304ba9b (DEFAULT_TABLE table=rebalance_test_table_8 [id=ddaeb399d1c84b0caa828ddf2bf3d0e8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:38.954613 5161 tablet_service.cc:1467] Processing CreateTablet for tablet faf43fec190c4830957d4e3ab304ba9b (DEFAULT_TABLE table=rebalance_test_table_8 [id=ddaeb399d1c84b0caa828ddf2bf3d0e8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:38.954731 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 86ffaa5522df469e9bb410872fae256d (DEFAULT_TABLE table=rebalance_test_table_8 [id=ddaeb399d1c84b0caa828ddf2bf3d0e8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:38.955080 5161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faf43fec190c4830957d4e3ab304ba9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:38.955073 5291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faf43fec190c4830957d4e3ab304ba9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:38.955171 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ffaa5522df469e9bb410872fae256d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:38.957506 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 3d668173d46d48ff840a535273009626 (DEFAULT_TABLE table=rebalance_test_table_8 [id=ddaeb399d1c84b0caa828ddf2bf3d0e8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:38.957901 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d668173d46d48ff840a535273009626. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:38.958637 5160 tablet_service.cc:1467] Processing CreateTablet for tablet 86ffaa5522df469e9bb410872fae256d (DEFAULT_TABLE table=rebalance_test_table_8 [id=ddaeb399d1c84b0caa828ddf2bf3d0e8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:38.959018 5160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ffaa5522df469e9bb410872fae256d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:38.959045 5030 tablet_service.cc:1467] Processing CreateTablet for tablet 86ffaa5522df469e9bb410872fae256d (DEFAULT_TABLE table=rebalance_test_table_8 [id=ddaeb399d1c84b0caa828ddf2bf3d0e8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:38.959468 5030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ffaa5522df469e9bb410872fae256d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:38.962359 5031 tablet_service.cc:1467] Processing CreateTablet for tablet 3d668173d46d48ff840a535273009626 (DEFAULT_TABLE table=rebalance_test_table_8 [id=ddaeb399d1c84b0caa828ddf2bf3d0e8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:38.962726 5031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d668173d46d48ff840a535273009626. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:38.965751 5292 tablet_service.cc:1467] Processing CreateTablet for tablet 3d668173d46d48ff840a535273009626 (DEFAULT_TABLE table=rebalance_test_table_8 [id=ddaeb399d1c84b0caa828ddf2bf3d0e8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:38.953861 5029 tablet_service.cc:1467] Processing CreateTablet for tablet faf43fec190c4830957d4e3ab304ba9b (DEFAULT_TABLE table=rebalance_test_table_8 [id=ddaeb399d1c84b0caa828ddf2bf3d0e8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:38.966346 5292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d668173d46d48ff840a535273009626. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:38.966446 5029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faf43fec190c4830957d4e3ab304ba9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:38.971128 6723 tablet_bootstrap.cc:492] T faf43fec190c4830957d4e3ab304ba9b P 9808124b81e14d1290ab0a6641a4b836: Bootstrap starting.
I20240501 08:13:38.973466 6723 tablet_bootstrap.cc:654] T faf43fec190c4830957d4e3ab304ba9b P 9808124b81e14d1290ab0a6641a4b836: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:38.975343 6722 tablet_bootstrap.cc:492] T 86ffaa5522df469e9bb410872fae256d P 0a952997a9fc4bfb8d7cbc56de6e0598: Bootstrap starting.
I20240501 08:13:38.977077 6722 tablet_bootstrap.cc:654] T 86ffaa5522df469e9bb410872fae256d P 0a952997a9fc4bfb8d7cbc56de6e0598: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:38.984175 6723 tablet_bootstrap.cc:492] T faf43fec190c4830957d4e3ab304ba9b P 9808124b81e14d1290ab0a6641a4b836: No bootstrap required, opened a new log
I20240501 08:13:38.984361 6723 ts_tablet_manager.cc:1367] T faf43fec190c4830957d4e3ab304ba9b P 9808124b81e14d1290ab0a6641a4b836: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.010s
I20240501 08:13:38.985255 6723 raft_consensus.cc:348] T faf43fec190c4830957d4e3ab304ba9b P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:38.985481 6723 raft_consensus.cc:374] T faf43fec190c4830957d4e3ab304ba9b P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:38.985570 6723 raft_consensus.cc:729] T faf43fec190c4830957d4e3ab304ba9b P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Initialized, Role: FOLLOWER
I20240501 08:13:38.985843 6723 consensus_queue.cc:260] T faf43fec190c4830957d4e3ab304ba9b P 9808124b81e14d1290ab0a6641a4b836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:38.986896 6723 ts_tablet_manager.cc:1398] T faf43fec190c4830957d4e3ab304ba9b P 9808124b81e14d1290ab0a6641a4b836: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:13:38.987298 6726 tablet_bootstrap.cc:492] T faf43fec190c4830957d4e3ab304ba9b P b6cc443a494e4595b0e7cf1777620a3f: Bootstrap starting.
I20240501 08:13:38.990365 6721 tablet_bootstrap.cc:492] T 86ffaa5522df469e9bb410872fae256d P 2c4d718738d1408eabce489f4e140af6: Bootstrap starting.
I20240501 08:13:38.990954 6726 tablet_bootstrap.cc:654] T faf43fec190c4830957d4e3ab304ba9b P b6cc443a494e4595b0e7cf1777620a3f: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:38.991202 6722 tablet_bootstrap.cc:492] T 86ffaa5522df469e9bb410872fae256d P 0a952997a9fc4bfb8d7cbc56de6e0598: No bootstrap required, opened a new log
I20240501 08:13:38.991398 6722 ts_tablet_manager.cc:1367] T 86ffaa5522df469e9bb410872fae256d P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.009s
I20240501 08:13:38.991772 6721 tablet_bootstrap.cc:654] T 86ffaa5522df469e9bb410872fae256d P 2c4d718738d1408eabce489f4e140af6: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:38.992166 6722 raft_consensus.cc:348] T 86ffaa5522df469e9bb410872fae256d P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:38.992370 6722 raft_consensus.cc:374] T 86ffaa5522df469e9bb410872fae256d P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:38.992456 6722 raft_consensus.cc:729] T 86ffaa5522df469e9bb410872fae256d P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Initialized, Role: FOLLOWER
I20240501 08:13:38.992779 6723 tablet_bootstrap.cc:492] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836: Bootstrap starting.
I20240501 08:13:38.992707 6722 consensus_queue.cc:260] T 86ffaa5522df469e9bb410872fae256d P 0a952997a9fc4bfb8d7cbc56de6e0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:38.994112 6722 ts_tablet_manager.cc:1398] T 86ffaa5522df469e9bb410872fae256d P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240501 08:13:38.994374 6722 tablet_bootstrap.cc:492] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598: Bootstrap starting.
I20240501 08:13:38.994364 6723 tablet_bootstrap.cc:654] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:38.995887 6722 tablet_bootstrap.cc:654] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:38.996426 6723 tablet_bootstrap.cc:492] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836: No bootstrap required, opened a new log
I20240501 08:13:38.996578 6723 ts_tablet_manager.cc:1367] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:13:38.997603 6723 raft_consensus.cc:348] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:38.997903 6723 raft_consensus.cc:374] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:38.998025 6723 raft_consensus.cc:729] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Initialized, Role: FOLLOWER
I20240501 08:13:38.998365 6723 consensus_queue.cc:260] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:38.999274 6723 ts_tablet_manager.cc:1398] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 08:13:39.001040 6721 tablet_bootstrap.cc:492] T 86ffaa5522df469e9bb410872fae256d P 2c4d718738d1408eabce489f4e140af6: No bootstrap required, opened a new log
I20240501 08:13:39.001235 6721 ts_tablet_manager.cc:1367] T 86ffaa5522df469e9bb410872fae256d P 2c4d718738d1408eabce489f4e140af6: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20240501 08:13:39.002271 6726 tablet_bootstrap.cc:492] T faf43fec190c4830957d4e3ab304ba9b P b6cc443a494e4595b0e7cf1777620a3f: No bootstrap required, opened a new log
I20240501 08:13:39.002440 6726 ts_tablet_manager.cc:1367] T faf43fec190c4830957d4e3ab304ba9b P b6cc443a494e4595b0e7cf1777620a3f: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240501 08:13:39.002704 6721 raft_consensus.cc:348] T 86ffaa5522df469e9bb410872fae256d P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:39.002921 6721 raft_consensus.cc:374] T 86ffaa5522df469e9bb410872fae256d P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:39.003019 6721 raft_consensus.cc:729] T 86ffaa5522df469e9bb410872fae256d P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Initialized, Role: FOLLOWER
I20240501 08:13:39.003309 6721 consensus_queue.cc:260] T 86ffaa5522df469e9bb410872fae256d P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:39.003454 6726 raft_consensus.cc:348] T faf43fec190c4830957d4e3ab304ba9b P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:39.003682 6726 raft_consensus.cc:374] T faf43fec190c4830957d4e3ab304ba9b P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:39.003769 6726 raft_consensus.cc:729] T faf43fec190c4830957d4e3ab304ba9b P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6cc443a494e4595b0e7cf1777620a3f, State: Initialized, Role: FOLLOWER
I20240501 08:13:39.003954 6721 ts_tablet_manager.cc:1398] T 86ffaa5522df469e9bb410872fae256d P 2c4d718738d1408eabce489f4e140af6: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240501 08:13:39.003988 6726 consensus_queue.cc:260] T faf43fec190c4830957d4e3ab304ba9b P b6cc443a494e4595b0e7cf1777620a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:39.004833 6721 tablet_bootstrap.cc:492] T 3d668173d46d48ff840a535273009626 P 2c4d718738d1408eabce489f4e140af6: Bootstrap starting.
I20240501 08:13:39.005084 6726 ts_tablet_manager.cc:1398] T faf43fec190c4830957d4e3ab304ba9b P b6cc443a494e4595b0e7cf1777620a3f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:39.005441 6726 tablet_bootstrap.cc:492] T 3d668173d46d48ff840a535273009626 P b6cc443a494e4595b0e7cf1777620a3f: Bootstrap starting.
I20240501 08:13:39.006950 6721 tablet_bootstrap.cc:654] T 3d668173d46d48ff840a535273009626 P 2c4d718738d1408eabce489f4e140af6: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:39.007373 6726 tablet_bootstrap.cc:654] T 3d668173d46d48ff840a535273009626 P b6cc443a494e4595b0e7cf1777620a3f: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:39.009418 6721 tablet_bootstrap.cc:492] T 3d668173d46d48ff840a535273009626 P 2c4d718738d1408eabce489f4e140af6: No bootstrap required, opened a new log
I20240501 08:13:39.009609 6721 ts_tablet_manager.cc:1367] T 3d668173d46d48ff840a535273009626 P 2c4d718738d1408eabce489f4e140af6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240501 08:13:39.010413 6721 raft_consensus.cc:348] T 3d668173d46d48ff840a535273009626 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:39.010668 6721 raft_consensus.cc:374] T 3d668173d46d48ff840a535273009626 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:39.010752 6721 raft_consensus.cc:729] T 3d668173d46d48ff840a535273009626 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Initialized, Role: FOLLOWER
I20240501 08:13:39.011034 6726 tablet_bootstrap.cc:492] T 3d668173d46d48ff840a535273009626 P b6cc443a494e4595b0e7cf1777620a3f: No bootstrap required, opened a new log
I20240501 08:13:39.011018 6721 consensus_queue.cc:260] T 3d668173d46d48ff840a535273009626 P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:39.011194 6726 ts_tablet_manager.cc:1367] T 3d668173d46d48ff840a535273009626 P b6cc443a494e4595b0e7cf1777620a3f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20240501 08:13:39.011595 6721 ts_tablet_manager.cc:1398] T 3d668173d46d48ff840a535273009626 P 2c4d718738d1408eabce489f4e140af6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:13:39.011910 6726 raft_consensus.cc:348] T 3d668173d46d48ff840a535273009626 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:39.012079 6726 raft_consensus.cc:374] T 3d668173d46d48ff840a535273009626 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:39.012157 6726 raft_consensus.cc:729] T 3d668173d46d48ff840a535273009626 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6cc443a494e4595b0e7cf1777620a3f, State: Initialized, Role: FOLLOWER
I20240501 08:13:39.012399 6726 consensus_queue.cc:260] T 3d668173d46d48ff840a535273009626 P b6cc443a494e4595b0e7cf1777620a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:39.012923 6726 ts_tablet_manager.cc:1398] T 3d668173d46d48ff840a535273009626 P b6cc443a494e4595b0e7cf1777620a3f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:13:39.013274 6726 tablet_bootstrap.cc:492] T 86ffaa5522df469e9bb410872fae256d P b6cc443a494e4595b0e7cf1777620a3f: Bootstrap starting.
I20240501 08:13:39.014392 6722 tablet_bootstrap.cc:492] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598: No bootstrap required, opened a new log
I20240501 08:13:39.014568 6722 ts_tablet_manager.cc:1367] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.004s
I20240501 08:13:39.014856 6726 tablet_bootstrap.cc:654] T 86ffaa5522df469e9bb410872fae256d P b6cc443a494e4595b0e7cf1777620a3f: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:39.015403 6722 raft_consensus.cc:348] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:39.015610 6722 raft_consensus.cc:374] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:39.015692 6722 raft_consensus.cc:729] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Initialized, Role: FOLLOWER
I20240501 08:13:39.015959 6722 consensus_queue.cc:260] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:39.016820 6722 ts_tablet_manager.cc:1398] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:39.019042 6726 tablet_bootstrap.cc:492] T 86ffaa5522df469e9bb410872fae256d P b6cc443a494e4595b0e7cf1777620a3f: No bootstrap required, opened a new log
I20240501 08:13:39.019197 6726 ts_tablet_manager.cc:1367] T 86ffaa5522df469e9bb410872fae256d P b6cc443a494e4595b0e7cf1777620a3f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 08:13:39.019767 6726 raft_consensus.cc:348] T 86ffaa5522df469e9bb410872fae256d P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:39.019913 6726 raft_consensus.cc:374] T 86ffaa5522df469e9bb410872fae256d P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:39.020001 6726 raft_consensus.cc:729] T 86ffaa5522df469e9bb410872fae256d P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6cc443a494e4595b0e7cf1777620a3f, State: Initialized, Role: FOLLOWER
I20240501 08:13:39.020267 6726 consensus_queue.cc:260] T 86ffaa5522df469e9bb410872fae256d P b6cc443a494e4595b0e7cf1777620a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:39.020720 6726 ts_tablet_manager.cc:1398] T 86ffaa5522df469e9bb410872fae256d P b6cc443a494e4595b0e7cf1777620a3f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 08:13:39.095845 6528 raft_consensus.cc:482] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:39.096072 6528 raft_consensus.cc:504] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:39.096808 6528 leader_election.cc:288] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797), b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417)
I20240501 08:13:39.097316 5439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d668173d46d48ff840a535273009626" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4d718738d1408eabce489f4e140af6" is_pre_election: true
I20240501 08:13:39.097342 5049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d668173d46d48ff840a535273009626" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f" is_pre_election: true
I20240501 08:13:39.097544 5439 raft_consensus.cc:2454] T 3d668173d46d48ff840a535273009626 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 0.
I20240501 08:13:39.097560 5049 raft_consensus.cc:2454] T 3d668173d46d48ff840a535273009626 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 0.
I20240501 08:13:39.098011 5248 leader_election.cc:302] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4d718738d1408eabce489f4e140af6, 9808124b81e14d1290ab0a6641a4b836; no voters:
I20240501 08:13:39.098255 6528 raft_consensus.cc:2781] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:39.098342 6528 raft_consensus.cc:482] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:39.098440 6528 raft_consensus.cc:3037] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:39.100728 6528 raft_consensus.cc:504] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:39.101357 6528 leader_election.cc:288] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 1 election: Requested vote from peers 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797), b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417)
I20240501 08:13:39.101778 5439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d668173d46d48ff840a535273009626" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:39.102017 5439 raft_consensus.cc:3037] T 3d668173d46d48ff840a535273009626 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:39.102015 5049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d668173d46d48ff840a535273009626" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:39.102138 6631 raft_consensus.cc:482] T 86ffaa5522df469e9bb410872fae256d P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:39.102241 5049 raft_consensus.cc:3037] T 3d668173d46d48ff840a535273009626 P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:39.102304 6631 raft_consensus.cc:504] T 86ffaa5522df469e9bb410872fae256d P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:39.103037 6631 leader_election.cc:288] T 86ffaa5522df469e9bb410872fae256d P b6cc443a494e4595b0e7cf1777620a3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797), 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937)
I20240501 08:13:39.103456 5440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ffaa5522df469e9bb410872fae256d" candidate_uuid: "b6cc443a494e4595b0e7cf1777620a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4d718738d1408eabce489f4e140af6" is_pre_election: true
I20240501 08:13:39.103657 5440 raft_consensus.cc:2454] T 86ffaa5522df469e9bb410872fae256d P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6cc443a494e4595b0e7cf1777620a3f in term 0.
I20240501 08:13:39.104035 4987 leader_election.cc:302] T 86ffaa5522df469e9bb410872fae256d P b6cc443a494e4595b0e7cf1777620a3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4d718738d1408eabce489f4e140af6, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:39.104331 6631 raft_consensus.cc:2781] T 86ffaa5522df469e9bb410872fae256d P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:39.104421 6631 raft_consensus.cc:482] T 86ffaa5522df469e9bb410872fae256d P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:39.104522 6631 raft_consensus.cc:3037] T 86ffaa5522df469e9bb410872fae256d P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:39.104777 5439 raft_consensus.cc:2454] T 3d668173d46d48ff840a535273009626 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 1.
I20240501 08:13:39.104821 5049 raft_consensus.cc:2454] T 3d668173d46d48ff840a535273009626 P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 1.
I20240501 08:13:39.105237 5248 leader_election.cc:302] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4d718738d1408eabce489f4e140af6, 9808124b81e14d1290ab0a6641a4b836; no voters:
I20240501 08:13:39.105208 5181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ffaa5522df469e9bb410872fae256d" candidate_uuid: "b6cc443a494e4595b0e7cf1777620a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" is_pre_election: true
I20240501 08:13:39.105420 5181 raft_consensus.cc:2454] T 86ffaa5522df469e9bb410872fae256d P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6cc443a494e4595b0e7cf1777620a3f in term 0.
I20240501 08:13:39.105732 6528 raft_consensus.cc:2781] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:39.106020 6528 raft_consensus.cc:686] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836 [term 1 LEADER]: Becoming Leader. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:39.106304 6528 consensus_queue.cc:237] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:39.107018 6631 raft_consensus.cc:504] T 86ffaa5522df469e9bb410872fae256d P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:39.107949 5439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ffaa5522df469e9bb410872fae256d" candidate_uuid: "b6cc443a494e4595b0e7cf1777620a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:39.108137 5439 raft_consensus.cc:3037] T 86ffaa5522df469e9bb410872fae256d P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:39.108273 5181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ffaa5522df469e9bb410872fae256d" candidate_uuid: "b6cc443a494e4595b0e7cf1777620a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:39.108474 5181 raft_consensus.cc:3037] T 86ffaa5522df469e9bb410872fae256d P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:39.109915 5439 raft_consensus.cc:2454] T 86ffaa5522df469e9bb410872fae256d P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6cc443a494e4595b0e7cf1777620a3f in term 1.
I20240501 08:13:39.110060 6631 leader_election.cc:288] T 86ffaa5522df469e9bb410872fae256d P b6cc443a494e4595b0e7cf1777620a3f [CANDIDATE]: Term 1 election: Requested vote from peers 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797), 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937)
I20240501 08:13:39.109889 4907 catalog_manager.cc:5484] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131). New cstate: current_term: 1 leader_uuid: "9808124b81e14d1290ab0a6641a4b836" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:39.110399 4987 leader_election.cc:302] T 86ffaa5522df469e9bb410872fae256d P b6cc443a494e4595b0e7cf1777620a3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4d718738d1408eabce489f4e140af6, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:39.110661 6631 raft_consensus.cc:2781] T 86ffaa5522df469e9bb410872fae256d P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:39.110795 6631 raft_consensus.cc:686] T 86ffaa5522df469e9bb410872fae256d P b6cc443a494e4595b0e7cf1777620a3f [term 1 LEADER]: Becoming Leader. State: Replica: b6cc443a494e4595b0e7cf1777620a3f, State: Running, Role: LEADER
I20240501 08:13:39.111033 6631 consensus_queue.cc:237] T 86ffaa5522df469e9bb410872fae256d P b6cc443a494e4595b0e7cf1777620a3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:39.111176 5181 raft_consensus.cc:2454] T 86ffaa5522df469e9bb410872fae256d P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6cc443a494e4595b0e7cf1777620a3f in term 1.
I20240501 08:13:39.114079 4904 catalog_manager.cc:5484] T 86ffaa5522df469e9bb410872fae256d P b6cc443a494e4595b0e7cf1777620a3f reported cstate change: term changed from 0 to 1, leader changed from <none> to b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129). New cstate: current_term: 1 leader_uuid: "b6cc443a494e4595b0e7cf1777620a3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:39.162559 6549 raft_consensus.cc:482] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:39.162750 6549 raft_consensus.cc:504] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:39.163401 6549 leader_election.cc:288] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417), 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983)
I20240501 08:13:39.163847 5049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf43fec190c4830957d4e3ab304ba9b" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f" is_pre_election: true
I20240501 08:13:39.163947 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf43fec190c4830957d4e3ab304ba9b" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808124b81e14d1290ab0a6641a4b836" is_pre_election: true
I20240501 08:13:39.164067 5049 raft_consensus.cc:2454] T faf43fec190c4830957d4e3ab304ba9b P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 0.
I20240501 08:13:39.164212 5311 raft_consensus.cc:2454] T faf43fec190c4830957d4e3ab304ba9b P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 0.
I20240501 08:13:39.164542 5117 leader_election.cc:302] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:39.164858 6549 raft_consensus.cc:2781] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:39.164970 6549 raft_consensus.cc:482] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:39.165076 6549 raft_consensus.cc:3037] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:39.167294 6549 raft_consensus.cc:504] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:39.168016 6549 leader_election.cc:288] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 election: Requested vote from peers b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417), 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983)
I20240501 08:13:39.168381 5049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf43fec190c4830957d4e3ab304ba9b" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:39.168640 5049 raft_consensus.cc:3037] T faf43fec190c4830957d4e3ab304ba9b P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:39.168572 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf43fec190c4830957d4e3ab304ba9b" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
I20240501 08:13:39.168823 5311 raft_consensus.cc:3037] T faf43fec190c4830957d4e3ab304ba9b P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:39.171627 5311 raft_consensus.cc:2454] T faf43fec190c4830957d4e3ab304ba9b P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 1.
I20240501 08:13:39.171620 5049 raft_consensus.cc:2454] T faf43fec190c4830957d4e3ab304ba9b P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 1.
I20240501 08:13:39.172082 5117 leader_election.cc:302] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:39.172355 6549 raft_consensus.cc:2781] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:39.172534 6549 raft_consensus.cc:686] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Becoming Leader. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Running, Role: LEADER
I20240501 08:13:39.172784 6549 consensus_queue.cc:237] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:39.175797 4904 catalog_manager.cc:5484] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130). New cstate: current_term: 1 leader_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:39.550361 6406 consensus_queue.cc:1035] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:39.567508 6406 consensus_queue.cc:1035] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 08:13:39.574735 6631 consensus_queue.cc:1035] T 86ffaa5522df469e9bb410872fae256d P b6cc443a494e4595b0e7cf1777620a3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:13:39.578612 6737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:39.579079 6737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 08:13:39.586994 6704 consensus_queue.cc:1035] T 86ffaa5522df469e9bb410872fae256d P b6cc443a494e4595b0e7cf1777620a3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:13:39.589148 6737 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:13:39.621497 6528 consensus_queue.cc:1035] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_8'
I20240501 08:13:39.629117 6757 consensus_queue.cc:1035] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
INSERT report
rows total: 2000
time total: 203.664 ms
time per row: 0.101832 ms
I20240501 08:13:39.961019 4906 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33308:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:13:39.970156 5422 tablet_service.cc:1467] Processing CreateTablet for tablet f8f72bb32e34420eab300223e4428591 (DEFAULT_TABLE table=rebalance_test_table_9 [id=81d0bbf5a5be4332a9e9b685d0a3a187]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:39.970156 5292 tablet_service.cc:1467] Processing CreateTablet for tablet f8f72bb32e34420eab300223e4428591 (DEFAULT_TABLE table=rebalance_test_table_9 [id=81d0bbf5a5be4332a9e9b685d0a3a187]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:39.970402 5161 tablet_service.cc:1467] Processing CreateTablet for tablet f8f72bb32e34420eab300223e4428591 (DEFAULT_TABLE table=rebalance_test_table_9 [id=81d0bbf5a5be4332a9e9b685d0a3a187]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:39.970638 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8f72bb32e34420eab300223e4428591. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:39.970638 5292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8f72bb32e34420eab300223e4428591. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:39.970830 5161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8f72bb32e34420eab300223e4428591. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:39.973767 5162 tablet_service.cc:1467] Processing CreateTablet for tablet 5e1018f0fee24311b58936ace2f6d53b (DEFAULT_TABLE table=rebalance_test_table_9 [id=81d0bbf5a5be4332a9e9b685d0a3a187]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:39.973855 5160 tablet_service.cc:1467] Processing CreateTablet for tablet 19585175235c4877b9c4da931c345565 (DEFAULT_TABLE table=rebalance_test_table_9 [id=81d0bbf5a5be4332a9e9b685d0a3a187]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:39.974164 5162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e1018f0fee24311b58936ace2f6d53b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:39.977342 5160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19585175235c4877b9c4da931c345565. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:39.977250 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 19585175235c4877b9c4da931c345565 (DEFAULT_TABLE table=rebalance_test_table_9 [id=81d0bbf5a5be4332a9e9b685d0a3a187]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:39.977733 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19585175235c4877b9c4da931c345565. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:39.984993 5029 tablet_service.cc:1467] Processing CreateTablet for tablet 5e1018f0fee24311b58936ace2f6d53b (DEFAULT_TABLE table=rebalance_test_table_9 [id=81d0bbf5a5be4332a9e9b685d0a3a187]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:39.985464 5029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e1018f0fee24311b58936ace2f6d53b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:39.985527 5291 tablet_service.cc:1467] Processing CreateTablet for tablet 19585175235c4877b9c4da931c345565 (DEFAULT_TABLE table=rebalance_test_table_9 [id=81d0bbf5a5be4332a9e9b685d0a3a187]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:39.986052 5291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19585175235c4877b9c4da931c345565. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:39.987789 6803 tablet_bootstrap.cc:492] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598: Bootstrap starting.
I20240501 08:13:39.988936 6802 tablet_bootstrap.cc:492] T f8f72bb32e34420eab300223e4428591 P 2c4d718738d1408eabce489f4e140af6: Bootstrap starting.
I20240501 08:13:39.989571 6803 tablet_bootstrap.cc:654] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:39.990407 5290 tablet_service.cc:1467] Processing CreateTablet for tablet 5e1018f0fee24311b58936ace2f6d53b (DEFAULT_TABLE table=rebalance_test_table_9 [id=81d0bbf5a5be4332a9e9b685d0a3a187]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:13:39.990649 6802 tablet_bootstrap.cc:654] T f8f72bb32e34420eab300223e4428591 P 2c4d718738d1408eabce489f4e140af6: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:39.990810 5290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e1018f0fee24311b58936ace2f6d53b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:39.995260 6801 tablet_bootstrap.cc:492] T f8f72bb32e34420eab300223e4428591 P 9808124b81e14d1290ab0a6641a4b836: Bootstrap starting.
I20240501 08:13:39.996628 6802 tablet_bootstrap.cc:492] T f8f72bb32e34420eab300223e4428591 P 2c4d718738d1408eabce489f4e140af6: No bootstrap required, opened a new log
I20240501 08:13:39.996873 6802 ts_tablet_manager.cc:1367] T f8f72bb32e34420eab300223e4428591 P 2c4d718738d1408eabce489f4e140af6: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20240501 08:13:39.997888 6801 tablet_bootstrap.cc:654] T f8f72bb32e34420eab300223e4428591 P 9808124b81e14d1290ab0a6641a4b836: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:39.998216 6802 raft_consensus.cc:348] T f8f72bb32e34420eab300223e4428591 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:39.998466 6802 raft_consensus.cc:374] T f8f72bb32e34420eab300223e4428591 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:39.998669 6802 raft_consensus.cc:729] T f8f72bb32e34420eab300223e4428591 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Initialized, Role: FOLLOWER
I20240501 08:13:39.998945 6802 consensus_queue.cc:260] T f8f72bb32e34420eab300223e4428591 P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:39.999481 6802 ts_tablet_manager.cc:1398] T f8f72bb32e34420eab300223e4428591 P 2c4d718738d1408eabce489f4e140af6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:13:40.000458 6802 tablet_bootstrap.cc:492] T 19585175235c4877b9c4da931c345565 P 2c4d718738d1408eabce489f4e140af6: Bootstrap starting.
I20240501 08:13:40.000952 6803 tablet_bootstrap.cc:492] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598: No bootstrap required, opened a new log
I20240501 08:13:40.001194 6803 ts_tablet_manager.cc:1367] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.004s
I20240501 08:13:40.002071 6803 raft_consensus.cc:348] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:40.002342 6803 raft_consensus.cc:374] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:40.002482 6803 raft_consensus.cc:729] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Initialized, Role: FOLLOWER
I20240501 08:13:40.002732 6803 consensus_queue.cc:260] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:40.003429 6803 ts_tablet_manager.cc:1398] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:13:40.003734 6803 tablet_bootstrap.cc:492] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598: Bootstrap starting.
I20240501 08:13:40.005591 6803 tablet_bootstrap.cc:654] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:40.005657 6802 tablet_bootstrap.cc:654] T 19585175235c4877b9c4da931c345565 P 2c4d718738d1408eabce489f4e140af6: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:40.008575 6806 tablet_bootstrap.cc:492] T 5e1018f0fee24311b58936ace2f6d53b P b6cc443a494e4595b0e7cf1777620a3f: Bootstrap starting.
I20240501 08:13:40.011164 6801 tablet_bootstrap.cc:492] T f8f72bb32e34420eab300223e4428591 P 9808124b81e14d1290ab0a6641a4b836: No bootstrap required, opened a new log
I20240501 08:13:40.011456 6801 ts_tablet_manager.cc:1367] T f8f72bb32e34420eab300223e4428591 P 9808124b81e14d1290ab0a6641a4b836: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.003s
I20240501 08:13:40.012507 6801 raft_consensus.cc:348] T f8f72bb32e34420eab300223e4428591 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:40.012660 6806 tablet_bootstrap.cc:654] T 5e1018f0fee24311b58936ace2f6d53b P b6cc443a494e4595b0e7cf1777620a3f: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:40.012780 6801 raft_consensus.cc:374] T f8f72bb32e34420eab300223e4428591 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:40.012889 6801 raft_consensus.cc:729] T f8f72bb32e34420eab300223e4428591 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Initialized, Role: FOLLOWER
I20240501 08:13:40.013265 6803 tablet_bootstrap.cc:492] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598: No bootstrap required, opened a new log
I20240501 08:13:40.013159 6801 consensus_queue.cc:260] T f8f72bb32e34420eab300223e4428591 P 9808124b81e14d1290ab0a6641a4b836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:40.013437 6803 ts_tablet_manager.cc:1367] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20240501 08:13:40.013823 6801 ts_tablet_manager.cc:1398] T f8f72bb32e34420eab300223e4428591 P 9808124b81e14d1290ab0a6641a4b836: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:13:40.014384 6801 tablet_bootstrap.cc:492] T 19585175235c4877b9c4da931c345565 P 9808124b81e14d1290ab0a6641a4b836: Bootstrap starting.
I20240501 08:13:40.014448 6803 raft_consensus.cc:348] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:40.014746 6803 raft_consensus.cc:374] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:40.014840 6803 raft_consensus.cc:729] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Initialized, Role: FOLLOWER
I20240501 08:13:40.015235 6803 consensus_queue.cc:260] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:40.015859 6803 ts_tablet_manager.cc:1398] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:40.016489 6803 tablet_bootstrap.cc:492] T f8f72bb32e34420eab300223e4428591 P 0a952997a9fc4bfb8d7cbc56de6e0598: Bootstrap starting.
I20240501 08:13:40.017418 6801 tablet_bootstrap.cc:654] T 19585175235c4877b9c4da931c345565 P 9808124b81e14d1290ab0a6641a4b836: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:40.017573 6802 tablet_bootstrap.cc:492] T 19585175235c4877b9c4da931c345565 P 2c4d718738d1408eabce489f4e140af6: No bootstrap required, opened a new log
I20240501 08:13:40.017808 6802 ts_tablet_manager.cc:1367] T 19585175235c4877b9c4da931c345565 P 2c4d718738d1408eabce489f4e140af6: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.002s
I20240501 08:13:40.018146 6803 tablet_bootstrap.cc:654] T f8f72bb32e34420eab300223e4428591 P 0a952997a9fc4bfb8d7cbc56de6e0598: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:40.018676 6802 raft_consensus.cc:348] T 19585175235c4877b9c4da931c345565 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:40.018922 6802 raft_consensus.cc:374] T 19585175235c4877b9c4da931c345565 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:40.019012 6802 raft_consensus.cc:729] T 19585175235c4877b9c4da931c345565 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Initialized, Role: FOLLOWER
I20240501 08:13:40.019268 6802 consensus_queue.cc:260] T 19585175235c4877b9c4da931c345565 P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:40.019826 6802 ts_tablet_manager.cc:1398] T 19585175235c4877b9c4da931c345565 P 2c4d718738d1408eabce489f4e140af6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:13:40.025180 6806 tablet_bootstrap.cc:492] T 5e1018f0fee24311b58936ace2f6d53b P b6cc443a494e4595b0e7cf1777620a3f: No bootstrap required, opened a new log
I20240501 08:13:40.025362 6806 ts_tablet_manager.cc:1367] T 5e1018f0fee24311b58936ace2f6d53b P b6cc443a494e4595b0e7cf1777620a3f: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20240501 08:13:40.025578 6801 tablet_bootstrap.cc:492] T 19585175235c4877b9c4da931c345565 P 9808124b81e14d1290ab0a6641a4b836: No bootstrap required, opened a new log
I20240501 08:13:40.025743 6801 ts_tablet_manager.cc:1367] T 19585175235c4877b9c4da931c345565 P 9808124b81e14d1290ab0a6641a4b836: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240501 08:13:40.026312 6806 raft_consensus.cc:348] T 5e1018f0fee24311b58936ace2f6d53b P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:40.026537 6806 raft_consensus.cc:374] T 5e1018f0fee24311b58936ace2f6d53b P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:40.026679 6806 raft_consensus.cc:729] T 5e1018f0fee24311b58936ace2f6d53b P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6cc443a494e4595b0e7cf1777620a3f, State: Initialized, Role: FOLLOWER
I20240501 08:13:40.026656 6801 raft_consensus.cc:348] T 19585175235c4877b9c4da931c345565 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:40.026873 6801 raft_consensus.cc:374] T 19585175235c4877b9c4da931c345565 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:40.026949 6801 raft_consensus.cc:729] T 19585175235c4877b9c4da931c345565 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Initialized, Role: FOLLOWER
I20240501 08:13:40.026983 6806 consensus_queue.cc:260] T 5e1018f0fee24311b58936ace2f6d53b P b6cc443a494e4595b0e7cf1777620a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:40.027252 6803 tablet_bootstrap.cc:492] T f8f72bb32e34420eab300223e4428591 P 0a952997a9fc4bfb8d7cbc56de6e0598: No bootstrap required, opened a new log
I20240501 08:13:40.027189 6801 consensus_queue.cc:260] T 19585175235c4877b9c4da931c345565 P 9808124b81e14d1290ab0a6641a4b836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:40.027449 6803 ts_tablet_manager.cc:1367] T f8f72bb32e34420eab300223e4428591 P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.006s
I20240501 08:13:40.027606 6806 ts_tablet_manager.cc:1398] T 5e1018f0fee24311b58936ace2f6d53b P b6cc443a494e4595b0e7cf1777620a3f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:40.028424 6803 raft_consensus.cc:348] T f8f72bb32e34420eab300223e4428591 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:40.028642 6803 raft_consensus.cc:374] T f8f72bb32e34420eab300223e4428591 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:40.028739 6803 raft_consensus.cc:729] T f8f72bb32e34420eab300223e4428591 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Initialized, Role: FOLLOWER
I20240501 08:13:40.029161 6803 consensus_queue.cc:260] T f8f72bb32e34420eab300223e4428591 P 0a952997a9fc4bfb8d7cbc56de6e0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:40.029819 6803 ts_tablet_manager.cc:1398] T f8f72bb32e34420eab300223e4428591 P 0a952997a9fc4bfb8d7cbc56de6e0598: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:13:40.031085 6801 ts_tablet_manager.cc:1398] T 19585175235c4877b9c4da931c345565 P 9808124b81e14d1290ab0a6641a4b836: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 08:13:40.031477 6801 tablet_bootstrap.cc:492] T 5e1018f0fee24311b58936ace2f6d53b P 9808124b81e14d1290ab0a6641a4b836: Bootstrap starting.
I20240501 08:13:40.034421 6801 tablet_bootstrap.cc:654] T 5e1018f0fee24311b58936ace2f6d53b P 9808124b81e14d1290ab0a6641a4b836: Neither blocks nor log segments found. Creating new log.
I20240501 08:13:40.036617 6801 tablet_bootstrap.cc:492] T 5e1018f0fee24311b58936ace2f6d53b P 9808124b81e14d1290ab0a6641a4b836: No bootstrap required, opened a new log
I20240501 08:13:40.036746 6801 ts_tablet_manager.cc:1367] T 5e1018f0fee24311b58936ace2f6d53b P 9808124b81e14d1290ab0a6641a4b836: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 08:13:40.037287 6801 raft_consensus.cc:348] T 5e1018f0fee24311b58936ace2f6d53b P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:40.037460 6801 raft_consensus.cc:374] T 5e1018f0fee24311b58936ace2f6d53b P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:13:40.037542 6801 raft_consensus.cc:729] T 5e1018f0fee24311b58936ace2f6d53b P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Initialized, Role: FOLLOWER
I20240501 08:13:40.037757 6801 consensus_queue.cc:260] T 5e1018f0fee24311b58936ace2f6d53b P 9808124b81e14d1290ab0a6641a4b836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:40.038234 6801 ts_tablet_manager.cc:1398] T 5e1018f0fee24311b58936ace2f6d53b P 9808124b81e14d1290ab0a6641a4b836: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240501 08:13:40.073385 6792 raft_consensus.cc:482] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:40.073592 6792 raft_consensus.cc:504] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:40.074265 6792 leader_election.cc:288] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417), 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983)
I20240501 08:13:40.074981 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1018f0fee24311b58936ace2f6d53b" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808124b81e14d1290ab0a6641a4b836" is_pre_election: true
I20240501 08:13:40.074962 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1018f0fee24311b58936ace2f6d53b" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f" is_pre_election: true
I20240501 08:13:40.075212 5311 raft_consensus.cc:2454] T 5e1018f0fee24311b58936ace2f6d53b P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 0.
I20240501 08:13:40.075228 5051 raft_consensus.cc:2454] T 5e1018f0fee24311b58936ace2f6d53b P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 0.
I20240501 08:13:40.075628 5117 leader_election.cc:302] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 9808124b81e14d1290ab0a6641a4b836; no voters:
I20240501 08:13:40.075875 6792 raft_consensus.cc:2781] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:40.075991 6792 raft_consensus.cc:482] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:40.076088 6792 raft_consensus.cc:3037] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:40.078208 6792 raft_consensus.cc:504] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:40.078796 6792 leader_election.cc:288] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 election: Requested vote from peers b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417), 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983)
I20240501 08:13:40.079197 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1018f0fee24311b58936ace2f6d53b" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:40.079262 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1018f0fee24311b58936ace2f6d53b" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
I20240501 08:13:40.079429 5051 raft_consensus.cc:3037] T 5e1018f0fee24311b58936ace2f6d53b P b6cc443a494e4595b0e7cf1777620a3f [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:40.079465 5311 raft_consensus.cc:3037] T 5e1018f0fee24311b58936ace2f6d53b P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:40.081281 5051 raft_consensus.cc:2454] T 5e1018f0fee24311b58936ace2f6d53b P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 1.
I20240501 08:13:40.081380 5311 raft_consensus.cc:2454] T 5e1018f0fee24311b58936ace2f6d53b P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 1.
I20240501 08:13:40.081753 5117 leader_election.cc:302] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:40.082074 6792 raft_consensus.cc:2781] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:40.082203 6792 raft_consensus.cc:686] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Becoming Leader. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Running, Role: LEADER
I20240501 08:13:40.082461 6792 consensus_queue.cc:237] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:40.085666 4906 catalog_manager.cc:5484] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130). New cstate: current_term: 1 leader_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: HEALTHY } } }
I20240501 08:13:40.214843 6792 raft_consensus.cc:482] T f8f72bb32e34420eab300223e4428591 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:40.215080 6792 raft_consensus.cc:504] T f8f72bb32e34420eab300223e4428591 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:40.215812 6792 leader_election.cc:288] T f8f72bb32e34420eab300223e4428591 P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983), 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797)
I20240501 08:13:40.216403 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8f72bb32e34420eab300223e4428591" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808124b81e14d1290ab0a6641a4b836" is_pre_election: true
I20240501 08:13:40.216454 5440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8f72bb32e34420eab300223e4428591" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4d718738d1408eabce489f4e140af6" is_pre_election: true
I20240501 08:13:40.216691 5440 raft_consensus.cc:2454] T f8f72bb32e34420eab300223e4428591 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 0.
I20240501 08:13:40.216667 5311 raft_consensus.cc:2454] T f8f72bb32e34420eab300223e4428591 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 0.
I20240501 08:13:40.217162 5117 leader_election.cc:302] T f8f72bb32e34420eab300223e4428591 P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 9808124b81e14d1290ab0a6641a4b836; no voters:
I20240501 08:13:40.217449 6792 raft_consensus.cc:2781] T f8f72bb32e34420eab300223e4428591 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:40.217548 6792 raft_consensus.cc:482] T f8f72bb32e34420eab300223e4428591 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:40.217649 6792 raft_consensus.cc:3037] T f8f72bb32e34420eab300223e4428591 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:40.219568 6792 raft_consensus.cc:504] T f8f72bb32e34420eab300223e4428591 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:40.220124 6792 leader_election.cc:288] T f8f72bb32e34420eab300223e4428591 P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 election: Requested vote from peers 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983), 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797)
I20240501 08:13:40.220603 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8f72bb32e34420eab300223e4428591" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
I20240501 08:13:40.220640 5440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8f72bb32e34420eab300223e4428591" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:40.220808 5440 raft_consensus.cc:3037] T f8f72bb32e34420eab300223e4428591 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:40.220811 5311 raft_consensus.cc:3037] T f8f72bb32e34420eab300223e4428591 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:40.222725 5440 raft_consensus.cc:2454] T f8f72bb32e34420eab300223e4428591 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 1.
I20240501 08:13:40.222932 5311 raft_consensus.cc:2454] T f8f72bb32e34420eab300223e4428591 P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 1.
I20240501 08:13:40.223333 5118 leader_election.cc:302] T f8f72bb32e34420eab300223e4428591 P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 2c4d718738d1408eabce489f4e140af6; no voters:
I20240501 08:13:40.223663 6792 raft_consensus.cc:2781] T f8f72bb32e34420eab300223e4428591 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:40.223919 6792 raft_consensus.cc:686] T f8f72bb32e34420eab300223e4428591 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Becoming Leader. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Running, Role: LEADER
I20240501 08:13:40.224228 6792 consensus_queue.cc:237] T f8f72bb32e34420eab300223e4428591 P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:40.227488 4906 catalog_manager.cc:5484] T f8f72bb32e34420eab300223e4428591 P 0a952997a9fc4bfb8d7cbc56de6e0598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130). New cstate: current_term: 1 leader_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: HEALTHY } } }
I20240501 08:13:40.325511 6792 raft_consensus.cc:482] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:13:40.325721 6792 raft_consensus.cc:504] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:40.326475 6792 leader_election.cc:288] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983), 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797)
I20240501 08:13:40.327014 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19585175235c4877b9c4da931c345565" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808124b81e14d1290ab0a6641a4b836" is_pre_election: true
I20240501 08:13:40.327044 5440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19585175235c4877b9c4da931c345565" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4d718738d1408eabce489f4e140af6" is_pre_election: true
I20240501 08:13:40.327242 5311 raft_consensus.cc:2454] T 19585175235c4877b9c4da931c345565 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 0.
I20240501 08:13:40.327270 5440 raft_consensus.cc:2454] T 19585175235c4877b9c4da931c345565 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 0.
I20240501 08:13:40.327679 5117 leader_election.cc:302] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 9808124b81e14d1290ab0a6641a4b836; no voters:
I20240501 08:13:40.327991 6792 raft_consensus.cc:2781] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:13:40.328096 6792 raft_consensus.cc:482] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:13:40.328183 6792 raft_consensus.cc:3037] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:40.330216 6792 raft_consensus.cc:504] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:40.330782 6792 leader_election.cc:288] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 election: Requested vote from peers 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983), 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797)
I20240501 08:13:40.331315 5440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19585175235c4877b9c4da931c345565" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:40.331341 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19585175235c4877b9c4da931c345565" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
I20240501 08:13:40.331518 5311 raft_consensus.cc:3037] T 19585175235c4877b9c4da931c345565 P 9808124b81e14d1290ab0a6641a4b836 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:40.331528 5440 raft_consensus.cc:3037] T 19585175235c4877b9c4da931c345565 P 2c4d718738d1408eabce489f4e140af6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:13:40.333298 5440 raft_consensus.cc:2454] T 19585175235c4877b9c4da931c345565 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 1.
I20240501 08:13:40.333412 5311 raft_consensus.cc:2454] T 19585175235c4877b9c4da931c345565 P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 1.
I20240501 08:13:40.333796 5118 leader_election.cc:302] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 2c4d718738d1408eabce489f4e140af6; no voters:
I20240501 08:13:40.334146 6792 raft_consensus.cc:2781] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:13:40.334300 6792 raft_consensus.cc:686] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Becoming Leader. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Running, Role: LEADER
I20240501 08:13:40.334584 6792 consensus_queue.cc:237] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:40.337709 4906 catalog_manager.cc:5484] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130). New cstate: current_term: 1 leader_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:40.638665 6792 consensus_queue.cc:1035] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:40.652313 6792 consensus_queue.cc:1035] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:40.687253 6822 consensus_queue.cc:1035] T f8f72bb32e34420eab300223e4428591 P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:40.693745 6823 consensus_queue.cc:1035] T f8f72bb32e34420eab300223e4428591 P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:13:40.730180 6818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:40.730540 6818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:13:40.738063 6818 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240501 08:13:40.806577 6791 consensus_queue.cc:1035] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:13:40.810286 6791 consensus_queue.cc:1035] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 212.784 ms
time per row: 0.106392 ms
I20240501 08:13:41.095175 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.133:32805
--local_ip_for_outbound_sockets=127.21.84.133
--tserver_master_addrs=127.21.84.190:46613
--webserver_port=40227
--webserver_interface=127.21.84.133
--builtin_ntp_servers=127.21.84.148:37541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 08:13:41.440219 6874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:41.440527 6874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:13:41.440624 6874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:13:41.448076 6874 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:13:41.448206 6874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:13:41.448324 6874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.133
I20240501 08:13:41.458354 6874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:37541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.133:32805
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.21.84.133
--webserver_port=40227
--tserver_master_addrs=127.21.84.190:46613
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.133
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:13:41.459710 6874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:13:41.461429 6874 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:13:41.474548 6881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:41.475108 6880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:41.475881 6883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:13:41.480756 6874 server_base.cc:1034] running on GCE node
I20240501 08:13:41.481405 6874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:13:41.482494 6874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:13:41.483747 6874 hybrid_clock.cc:648] HybridClock initialized: now 1714551221483681 us; error 51 us; skew 500 ppm
I20240501 08:13:41.484134 6874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:13:41.486497 6874 webserver.cc:460] Webserver started at http://127.21.84.133:40227/ using document root <none> and password file <none>
I20240501 08:13:41.487321 6874 fs_manager.cc:359] Metadata directory not provided
I20240501 08:13:41.487430 6874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:13:41.493139 6874 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 08:13:41.495929 6889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:41.497629 6874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 08:13:41.497784 6874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "6651d7356f5c48388f38c1034d466962"
format_stamp: "Formatted at 2024-05-01 08:13:23 on dist-test-slave-9cgk"
I20240501 08:13:41.498648 6874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:13:41.541174 6874 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:13:41.542187 6874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:13:41.542541 6874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:13:41.543704 6874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:13:41.545276 6874 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:13:41.545363 6874 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:41.545491 6874 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:13:41.545544 6874 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:41.588780 6874 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.133:32805
I20240501 08:13:41.588815 7001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.133:32805 every 8 connection(s)
I20240501 08:13:41.590488 6874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 08:13:41.592792 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 6874
I20240501 08:13:41.593592 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.134:39137
--local_ip_for_outbound_sockets=127.21.84.134
--tserver_master_addrs=127.21.84.190:46613
--webserver_port=36543
--webserver_interface=127.21.84.134
--builtin_ntp_servers=127.21.84.148:37541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:13:41.604143 7002 heartbeater.cc:347] Connected to a master server at 127.21.84.190:46613
I20240501 08:13:41.604476 7002 heartbeater.cc:464] Registering TS with master...
I20240501 08:13:41.605276 7002 heartbeater.cc:511] Master 127.21.84.190:46613 requested a full tablet report, sending...
I20240501 08:13:41.606907 4909 ts_manager.cc:194] Re-registered known tserver with Master: 6651d7356f5c48388f38c1034d466962 (127.21.84.133:32805)
I20240501 08:13:41.608105 4909 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.133:35989
W20240501 08:13:41.944128 7006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:41.944422 7006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:13:41.944532 7006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:13:41.952574 7006 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:13:41.952762 7006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:13:41.952944 7006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.134
I20240501 08:13:41.963488 7006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:37541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.134:39137
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.21.84.134
--webserver_port=36543
--tserver_master_addrs=127.21.84.190:46613
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.134
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:13:41.965147 7006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:13:41.967155 7006 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:13:41.980485 7013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:41.981103 7015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:41.982000 7012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:13:41.982759 7006 server_base.cc:1034] running on GCE node
I20240501 08:13:41.983405 7006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:13:41.984375 7006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:13:41.985599 7006 hybrid_clock.cc:648] HybridClock initialized: now 1714551221985551 us; error 45 us; skew 500 ppm
I20240501 08:13:41.985955 7006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:13:41.988302 7006 webserver.cc:460] Webserver started at http://127.21.84.134:36543/ using document root <none> and password file <none>
I20240501 08:13:41.989038 7006 fs_manager.cc:359] Metadata directory not provided
I20240501 08:13:41.989171 7006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:13:41.994999 7006 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20240501 08:13:41.997814 7021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:41.999243 7006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 08:13:41.999428 7006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "36c507d2fe0c4ede9556db4ab9b563b1"
format_stamp: "Formatted at 2024-05-01 08:13:23 on dist-test-slave-9cgk"
I20240501 08:13:42.000231 7006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:13:42.021828 7006 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:13:42.022684 7006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:13:42.022984 7006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:13:42.024061 7006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:13:42.025605 7006 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:13:42.025676 7006 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:42.025794 7006 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:13:42.025838 7006 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:42.061590 7006 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.134:39137
I20240501 08:13:42.061627 7134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.134:39137 every 8 connection(s)
I20240501 08:13:42.063040 7006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 08:13:42.072374 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 7006
I20240501 08:13:42.073004 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.135:44223
--local_ip_for_outbound_sockets=127.21.84.135
--tserver_master_addrs=127.21.84.190:46613
--webserver_port=46769
--webserver_interface=127.21.84.135
--builtin_ntp_servers=127.21.84.148:37541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:13:42.076197 7135 heartbeater.cc:347] Connected to a master server at 127.21.84.190:46613
I20240501 08:13:42.076570 7135 heartbeater.cc:464] Registering TS with master...
I20240501 08:13:42.077718 7135 heartbeater.cc:511] Master 127.21.84.190:46613 requested a full tablet report, sending...
I20240501 08:13:42.079401 4909 ts_manager.cc:194] Re-registered known tserver with Master: 36c507d2fe0c4ede9556db4ab9b563b1 (127.21.84.134:39137)
I20240501 08:13:42.080214 4909 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.134:52555
W20240501 08:13:42.423374 7138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:42.423663 7138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:13:42.423748 7138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:13:42.431318 7138 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:13:42.431457 7138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:13:42.431646 7138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.135
I20240501 08:13:42.441902 7138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:37541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.135:44223
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.21.84.135
--webserver_port=46769
--tserver_master_addrs=127.21.84.190:46613
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.135
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:13:42.443341 7138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:13:42.445091 7138 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:13:42.458236 7144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:42.459084 7145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:42.459568 7147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:13:42.461112 7138 server_base.cc:1034] running on GCE node
I20240501 08:13:42.461756 7138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:13:42.463057 7138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:13:42.464293 7138 hybrid_clock.cc:648] HybridClock initialized: now 1714551222464228 us; error 59 us; skew 500 ppm
I20240501 08:13:42.464623 7138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:13:42.466833 7138 webserver.cc:460] Webserver started at http://127.21.84.135:46769/ using document root <none> and password file <none>
I20240501 08:13:42.467574 7138 fs_manager.cc:359] Metadata directory not provided
I20240501 08:13:42.467679 7138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:13:42.473400 7138 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20240501 08:13:42.476187 7153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:42.477712 7138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 08:13:42.477880 7138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "eb21f6475035413984756905636d034f"
format_stamp: "Formatted at 2024-05-01 08:13:24 on dist-test-slave-9cgk"
I20240501 08:13:42.478868 7138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:13:42.505016 7138 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:13:42.505903 7138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:13:42.506270 7138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:13:42.507256 7138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:13:42.508622 7138 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:13:42.508697 7138 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:42.508795 7138 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:13:42.508833 7138 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:42.539355 7138 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.135:44223
I20240501 08:13:42.539386 7265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.135:44223 every 8 connection(s)
I20240501 08:13:42.540910 7138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 08:13:42.541502 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 7138
I20240501 08:13:42.542281 21842 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
/tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.84.136:37697
--local_ip_for_outbound_sockets=127.21.84.136
--tserver_master_addrs=127.21.84.190:46613
--webserver_port=43427
--webserver_interface=127.21.84.136
--builtin_ntp_servers=127.21.84.148:37541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 08:13:42.558020 7266 heartbeater.cc:347] Connected to a master server at 127.21.84.190:46613
I20240501 08:13:42.558300 7266 heartbeater.cc:464] Registering TS with master...
I20240501 08:13:42.559271 7266 heartbeater.cc:511] Master 127.21.84.190:46613 requested a full tablet report, sending...
I20240501 08:13:42.561548 4909 ts_manager.cc:194] Re-registered known tserver with Master: eb21f6475035413984756905636d034f (127.21.84.135:44223)
I20240501 08:13:42.562623 4909 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.135:52341
I20240501 08:13:42.610698 7002 heartbeater.cc:503] Master 127.21.84.190:46613 was elected leader, sending a full tablet report...
W20240501 08:13:42.903678 7269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:13:42.904057 7269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:13:42.904192 7269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:13:42.911579 7269 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 08:13:42.911724 7269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:13:42.911841 7269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.84.136
I20240501 08:13:42.921799 7269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.84.148:37541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.84.136:37697
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.21.84.136
--webserver_port=43427
--tserver_master_addrs=127.21.84.190:46613
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.84.136
--log_dir=/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:13:42.923215 7269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:13:42.925063 7269 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:13:42.938493 7275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:42.939110 7277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:13:42.940109 7274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:13:42.944058 7269 server_base.cc:1034] running on GCE node
I20240501 08:13:42.944728 7269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:13:42.945797 7269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:13:42.946998 7269 hybrid_clock.cc:648] HybridClock initialized: now 1714551222946948 us; error 48 us; skew 500 ppm
I20240501 08:13:42.947322 7269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:13:42.949554 7269 webserver.cc:460] Webserver started at http://127.21.84.136:43427/ using document root <none> and password file <none>
I20240501 08:13:42.950330 7269 fs_manager.cc:359] Metadata directory not provided
I20240501 08:13:42.950471 7269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:13:42.955858 7269 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 08:13:42.958467 7283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:42.959736 7269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:13:42.959888 7269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "67b454b1f4f344ebbfe96dced538e368"
format_stamp: "Formatted at 2024-05-01 08:13:24 on dist-test-slave-9cgk"
I20240501 08:13:42.960654 7269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:13:43.008065 7269 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:13:43.008935 7269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:13:43.009231 7269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:13:43.010208 7269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:13:43.011772 7269 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:13:43.011849 7269 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:43.011977 7269 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:13:43.012030 7269 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:13:43.048969 7269 rpc_server.cc:307] RPC server started. Bound to: 127.21.84.136:37697
I20240501 08:13:43.049047 7395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.84.136:37697 every 8 connection(s)
I20240501 08:13:43.050570 7269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHqQ3K9/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714551041067603-21842-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 08:13:43.055590 21842 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHqQ3K9/build/asan/bin/kudu as pid 7269
I20240501 08:13:43.072633 7396 heartbeater.cc:347] Connected to a master server at 127.21.84.190:46613
I20240501 08:13:43.073035 7396 heartbeater.cc:464] Registering TS with master...
I20240501 08:13:43.074072 7396 heartbeater.cc:511] Master 127.21.84.190:46613 requested a full tablet report, sending...
I20240501 08:13:43.075491 4909 ts_manager.cc:194] Re-registered known tserver with Master: 67b454b1f4f344ebbfe96dced538e368 (127.21.84.136:37697)
I20240501 08:13:43.076439 4909 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.84.136:49613
I20240501 08:13:43.083057 7135 heartbeater.cc:503] Master 127.21.84.190:46613 was elected leader, sending a full tablet report...
I20240501 08:13:43.100647 5439 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "19585175235c4877b9c4da931c345565"
dest_uuid: "2c4d718738d1408eabce489f4e140af6"
from {username='slave'} at 127.0.0.1:37632
I20240501 08:13:43.100911 5439 raft_consensus.cc:482] T 19585175235c4877b9c4da931c345565 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.101053 5439 raft_consensus.cc:3037] T 19585175235c4877b9c4da931c345565 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.103041 5439 raft_consensus.cc:504] T 19585175235c4877b9c4da931c345565 P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.103745 5439 leader_election.cc:288] T 19585175235c4877b9c4da931c345565 P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 2 election: Requested vote from peers 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937), 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983)
I20240501 08:13:43.104410 5181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19585175235c4877b9c4da931c345565" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:43.104434 5312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19585175235c4877b9c4da931c345565" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
I20240501 08:13:43.104624 5181 raft_consensus.cc:3032] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:43.104693 5181 raft_consensus.cc:729] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Running, Role: LEADER
I20240501 08:13:43.104641 5312 raft_consensus.cc:3037] T 19585175235c4877b9c4da931c345565 P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.104971 5181 consensus_queue.cc:260] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.105517 5181 raft_consensus.cc:3037] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.108345 5181 raft_consensus.cc:2454] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 2.
I20240501 08:13:43.108531 5312 raft_consensus.cc:2454] T 19585175235c4877b9c4da931c345565 P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 2.
I20240501 08:13:43.108937 5376 leader_election.cc:302] T 19585175235c4877b9c4da931c345565 P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 2c4d718738d1408eabce489f4e140af6; no voters:
I20240501 08:13:43.109964 7007 raft_consensus.cc:2781] T 19585175235c4877b9c4da931c345565 P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:43.110137 7007 raft_consensus.cc:686] T 19585175235c4877b9c4da931c345565 P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Becoming Leader. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:43.110503 7007 consensus_queue.cc:237] T 19585175235c4877b9c4da931c345565 P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.130143 4909 catalog_manager.cc:5484] T 19585175235c4877b9c4da931c345565 P 2c4d718738d1408eabce489f4e140af6 reported cstate change: term changed from 1 to 2, leader changed from 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130) to 2c4d718738d1408eabce489f4e140af6 (127.21.84.132). New cstate: current_term: 2 leader_uuid: "2c4d718738d1408eabce489f4e140af6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: HEALTHY } } }
I20240501 08:13:43.132928 5439 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3d668173d46d48ff840a535273009626"
dest_uuid: "2c4d718738d1408eabce489f4e140af6"
from {username='slave'} at 127.0.0.1:37632
I20240501 08:13:43.133203 5439 raft_consensus.cc:482] T 3d668173d46d48ff840a535273009626 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.133347 5439 raft_consensus.cc:3037] T 3d668173d46d48ff840a535273009626 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.135666 5439 raft_consensus.cc:504] T 3d668173d46d48ff840a535273009626 P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:43.136422 5439 leader_election.cc:288] T 3d668173d46d48ff840a535273009626 P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 2 election: Requested vote from peers 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983), b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417)
I20240501 08:13:43.137081 5312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d668173d46d48ff840a535273009626" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
I20240501 08:13:43.137356 5312 raft_consensus.cc:3032] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:43.137471 5312 raft_consensus.cc:729] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:43.137636 5439 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "86ffaa5522df469e9bb410872fae256d"
dest_uuid: "2c4d718738d1408eabce489f4e140af6"
from {username='slave'} at 127.0.0.1:37632
I20240501 08:13:43.137833 5439 raft_consensus.cc:482] T 86ffaa5522df469e9bb410872fae256d P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.137953 5439 raft_consensus.cc:3037] T 86ffaa5522df469e9bb410872fae256d P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.137828 5312 consensus_queue.cc:260] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:43.138499 5312 raft_consensus.cc:3037] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.140673 5439 raft_consensus.cc:504] T 86ffaa5522df469e9bb410872fae256d P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:43.141546 5439 leader_election.cc:288] T 86ffaa5522df469e9bb410872fae256d P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 2 election: Requested vote from peers b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417), 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937)
I20240501 08:13:43.141783 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ffaa5522df469e9bb410872fae256d" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:43.142028 5051 raft_consensus.cc:3032] T 86ffaa5522df469e9bb410872fae256d P b6cc443a494e4595b0e7cf1777620a3f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:43.142158 5051 raft_consensus.cc:729] T 86ffaa5522df469e9bb410872fae256d P b6cc443a494e4595b0e7cf1777620a3f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b6cc443a494e4595b0e7cf1777620a3f, State: Running, Role: LEADER
I20240501 08:13:43.142555 5051 consensus_queue.cc:260] T 86ffaa5522df469e9bb410872fae256d P b6cc443a494e4595b0e7cf1777620a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:43.142694 5439 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "19ec6a8b857d428d858fa831e825c578"
dest_uuid: "2c4d718738d1408eabce489f4e140af6"
from {username='slave'} at 127.0.0.1:37632
I20240501 08:13:43.142869 5439 raft_consensus.cc:482] T 19ec6a8b857d428d858fa831e825c578 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.142999 5439 raft_consensus.cc:3037] T 19ec6a8b857d428d858fa831e825c578 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.143182 5051 raft_consensus.cc:3037] T 86ffaa5522df469e9bb410872fae256d P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.145725 5051 raft_consensus.cc:2454] T 86ffaa5522df469e9bb410872fae256d P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 2.
I20240501 08:13:43.145722 5439 raft_consensus.cc:504] T 19ec6a8b857d428d858fa831e825c578 P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.146678 5181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ffaa5522df469e9bb410872fae256d" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:43.146720 5049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d668173d46d48ff840a535273009626" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:43.146831 5439 leader_election.cc:288] T 19ec6a8b857d428d858fa831e825c578 P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 2 election: Requested vote from peers b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417), 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983)
I20240501 08:13:43.146909 5181 raft_consensus.cc:3037] T 86ffaa5522df469e9bb410872fae256d P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.146930 5049 raft_consensus.cc:3037] T 3d668173d46d48ff840a535273009626 P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.147799 5439 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c1bd10f523444d668736a2ff5e02442c"
dest_uuid: "2c4d718738d1408eabce489f4e140af6"
from {username='slave'} at 127.0.0.1:37632
I20240501 08:13:43.148000 5439 raft_consensus.cc:465] T c1bd10f523444d668736a2ff5e02442c P 2c4d718738d1408eabce489f4e140af6 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 08:13:43.148823 5439 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2e0d1fd6767b454e9871e1bdec86caa8"
dest_uuid: "2c4d718738d1408eabce489f4e140af6"
from {username='slave'} at 127.0.0.1:37632
I20240501 08:13:43.148979 5439 raft_consensus.cc:482] T 2e0d1fd6767b454e9871e1bdec86caa8 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.149077 5439 raft_consensus.cc:3037] T 2e0d1fd6767b454e9871e1bdec86caa8 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.149518 5181 raft_consensus.cc:2454] T 86ffaa5522df469e9bb410872fae256d P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 2.
I20240501 08:13:43.150208 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ec6a8b857d428d858fa831e825c578" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
I20240501 08:13:43.150450 5311 raft_consensus.cc:3032] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:43.150555 5311 raft_consensus.cc:729] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:43.150854 5311 consensus_queue.cc:260] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.151226 5311 raft_consensus.cc:3037] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.151661 5439 raft_consensus.cc:504] T 2e0d1fd6767b454e9871e1bdec86caa8 P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.152449 5439 leader_election.cc:288] T 2e0d1fd6767b454e9871e1bdec86caa8 P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 2 election: Requested vote from peers b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417), 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937)
I20240501 08:13:43.153339 5440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a3445b99b36444bb811486be77100479"
dest_uuid: "2c4d718738d1408eabce489f4e140af6"
from {username='slave'} at 127.0.0.1:37632
I20240501 08:13:43.153555 5440 raft_consensus.cc:482] T a3445b99b36444bb811486be77100479 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.153682 5440 raft_consensus.cc:3037] T a3445b99b36444bb811486be77100479 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.154795 5311 raft_consensus.cc:2454] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 2.
I20240501 08:13:43.154868 5050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ec6a8b857d428d858fa831e825c578" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:43.155088 5050 raft_consensus.cc:3037] T 19ec6a8b857d428d858fa831e825c578 P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.155472 5376 leader_election.cc:302] T 86ffaa5522df469e9bb410872fae256d P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 2c4d718738d1408eabce489f4e140af6; no voters:
I20240501 08:13:43.156066 5181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0d1fd6767b454e9871e1bdec86caa8" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:43.156275 5181 raft_consensus.cc:3032] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:43.156361 5181 raft_consensus.cc:729] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Running, Role: LEADER
I20240501 08:13:43.156592 5181 consensus_queue.cc:260] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.156913 5181 raft_consensus.cc:3037] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.157313 5440 raft_consensus.cc:504] T a3445b99b36444bb811486be77100479 P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:43.157661 5050 raft_consensus.cc:2454] T 19ec6a8b857d428d858fa831e825c578 P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 2.
I20240501 08:13:43.158128 5048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0d1fd6767b454e9871e1bdec86caa8" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:43.158365 5048 raft_consensus.cc:3037] T 2e0d1fd6767b454e9871e1bdec86caa8 P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.159382 5181 raft_consensus.cc:2454] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 2.
I20240501 08:13:43.160661 5377 leader_election.cc:302] T 19ec6a8b857d428d858fa831e825c578 P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4d718738d1408eabce489f4e140af6, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:43.160836 5048 raft_consensus.cc:2454] T 2e0d1fd6767b454e9871e1bdec86caa8 P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 2.
I20240501 08:13:43.160866 5376 leader_election.cc:302] T 2e0d1fd6767b454e9871e1bdec86caa8 P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 2c4d718738d1408eabce489f4e140af6; no voters:
I20240501 08:13:43.161139 7405 raft_consensus.cc:2781] T 86ffaa5522df469e9bb410872fae256d P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:43.161271 7405 raft_consensus.cc:686] T 86ffaa5522df469e9bb410872fae256d P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Becoming Leader. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:43.161381 5440 leader_election.cc:288] T a3445b99b36444bb811486be77100479 P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 2 election: Requested vote from peers b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417), 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983)
I20240501 08:13:43.160225 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3445b99b36444bb811486be77100479" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
I20240501 08:13:43.161638 5309 raft_consensus.cc:3037] T a3445b99b36444bb811486be77100479 P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.161607 7405 consensus_queue.cc:237] T 86ffaa5522df469e9bb410872fae256d P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:43.149621 5049 raft_consensus.cc:2454] T 3d668173d46d48ff840a535273009626 P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 2.
I20240501 08:13:43.162627 7007 raft_consensus.cc:2781] T 19ec6a8b857d428d858fa831e825c578 P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:43.162600 5312 raft_consensus.cc:2454] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 2.
I20240501 08:13:43.163270 5377 leader_election.cc:302] T 3d668173d46d48ff840a535273009626 P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4d718738d1408eabce489f4e140af6, 9808124b81e14d1290ab0a6641a4b836; no voters:
I20240501 08:13:43.164410 5309 raft_consensus.cc:2454] T a3445b99b36444bb811486be77100479 P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 2.
I20240501 08:13:43.165798 7407 raft_consensus.cc:2781] T 2e0d1fd6767b454e9871e1bdec86caa8 P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:43.166137 7407 raft_consensus.cc:686] T 2e0d1fd6767b454e9871e1bdec86caa8 P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Becoming Leader. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:43.166584 7407 consensus_queue.cc:237] T 2e0d1fd6767b454e9871e1bdec86caa8 P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.167502 5050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3445b99b36444bb811486be77100479" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:43.167748 5050 raft_consensus.cc:3032] T a3445b99b36444bb811486be77100479 P b6cc443a494e4595b0e7cf1777620a3f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:43.167856 5050 raft_consensus.cc:729] T a3445b99b36444bb811486be77100479 P b6cc443a494e4595b0e7cf1777620a3f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b6cc443a494e4595b0e7cf1777620a3f, State: Running, Role: LEADER
I20240501 08:13:43.167968 7404 raft_consensus.cc:2781] T 3d668173d46d48ff840a535273009626 P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:43.168112 7404 raft_consensus.cc:686] T 3d668173d46d48ff840a535273009626 P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Becoming Leader. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:43.168233 5050 consensus_queue.cc:260] T a3445b99b36444bb811486be77100479 P b6cc443a494e4595b0e7cf1777620a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:43.168409 7404 consensus_queue.cc:237] T 3d668173d46d48ff840a535273009626 P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:43.168787 5050 raft_consensus.cc:3037] T a3445b99b36444bb811486be77100479 P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.172703 5440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "14d9bd97e4674e0ebdd90bf45b5679cb"
dest_uuid: "2c4d718738d1408eabce489f4e140af6"
from {username='slave'} at 127.0.0.1:37632
I20240501 08:13:43.173058 5440 raft_consensus.cc:482] T 14d9bd97e4674e0ebdd90bf45b5679cb P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.173259 5440 raft_consensus.cc:3037] T 14d9bd97e4674e0ebdd90bf45b5679cb P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.185391 5050 raft_consensus.cc:2454] T a3445b99b36444bb811486be77100479 P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 2.
I20240501 08:13:43.176599 5440 raft_consensus.cc:504] T 14d9bd97e4674e0ebdd90bf45b5679cb P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.190786 5440 leader_election.cc:288] T 14d9bd97e4674e0ebdd90bf45b5679cb P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 2 election: Requested vote from peers 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983), b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417)
I20240501 08:13:43.192271 5439 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "416ef8d814614a7caba4279e64893b27"
dest_uuid: "2c4d718738d1408eabce489f4e140af6"
from {username='slave'} at 127.0.0.1:37632
I20240501 08:13:43.192548 5439 raft_consensus.cc:482] T 416ef8d814614a7caba4279e64893b27 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.192709 5439 raft_consensus.cc:3037] T 416ef8d814614a7caba4279e64893b27 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.196600 5439 raft_consensus.cc:504] T 416ef8d814614a7caba4279e64893b27 P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:43.197868 5439 leader_election.cc:288] T 416ef8d814614a7caba4279e64893b27 P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 2 election: Requested vote from peers b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417), 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937)
I20240501 08:13:43.198284 5181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "416ef8d814614a7caba4279e64893b27" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:43.198520 5181 raft_consensus.cc:3032] T 416ef8d814614a7caba4279e64893b27 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:43.198617 5181 raft_consensus.cc:729] T 416ef8d814614a7caba4279e64893b27 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Running, Role: LEADER
I20240501 08:13:43.198879 5181 consensus_queue.cc:260] T 416ef8d814614a7caba4279e64893b27 P 0a952997a9fc4bfb8d7cbc56de6e0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:43.199249 5181 raft_consensus.cc:3037] T 416ef8d814614a7caba4279e64893b27 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.202236 5181 raft_consensus.cc:2454] T 416ef8d814614a7caba4279e64893b27 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 2.
I20240501 08:13:43.202643 5441 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e9e30e05c0ef498288f35706978c981b"
dest_uuid: "2c4d718738d1408eabce489f4e140af6"
from {username='slave'} at 127.0.0.1:37632
I20240501 08:13:43.202885 5441 raft_consensus.cc:482] T e9e30e05c0ef498288f35706978c981b P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.203007 5441 raft_consensus.cc:3037] T e9e30e05c0ef498288f35706978c981b P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.206393 5376 leader_election.cc:302] T 416ef8d814614a7caba4279e64893b27 P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 2c4d718738d1408eabce489f4e140af6; no voters:
I20240501 08:13:43.207082 5441 raft_consensus.cc:504] T e9e30e05c0ef498288f35706978c981b P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.208369 5441 leader_election.cc:288] T e9e30e05c0ef498288f35706978c981b P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 2 election: Requested vote from peers 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937), b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417)
I20240501 08:13:43.215471 5440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8a96986cf35c4720991739af0626e1e6"
dest_uuid: "2c4d718738d1408eabce489f4e140af6"
from {username='slave'} at 127.0.0.1:37632
I20240501 08:13:43.216197 5440 raft_consensus.cc:465] T 8a96986cf35c4720991739af0626e1e6 P 2c4d718738d1408eabce489f4e140af6 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 08:13:43.217749 5441 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ab68600377eb4581a724b714cfe234d7"
dest_uuid: "2c4d718738d1408eabce489f4e140af6"
from {username='slave'} at 127.0.0.1:37632
I20240501 08:13:43.218111 5441 raft_consensus.cc:465] T ab68600377eb4581a724b714cfe234d7 P 2c4d718738d1408eabce489f4e140af6 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 08:13:43.219259 5439 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d381c4733f234e59b8eb6cf44166e007"
dest_uuid: "2c4d718738d1408eabce489f4e140af6"
from {username='slave'} at 127.0.0.1:37632
I20240501 08:13:43.219561 5439 raft_consensus.cc:465] T d381c4733f234e59b8eb6cf44166e007 P 2c4d718738d1408eabce489f4e140af6 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 08:13:43.220585 5441 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ff3b86014f2940b18707803c38bb5b8f"
dest_uuid: "2c4d718738d1408eabce489f4e140af6"
from {username='slave'} at 127.0.0.1:37632
I20240501 08:13:43.220830 5441 raft_consensus.cc:482] T ff3b86014f2940b18707803c38bb5b8f P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.221062 5441 raft_consensus.cc:3037] T ff3b86014f2940b18707803c38bb5b8f P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.224753 5441 raft_consensus.cc:504] T ff3b86014f2940b18707803c38bb5b8f P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.227312 5441 leader_election.cc:288] T ff3b86014f2940b18707803c38bb5b8f P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 2 election: Requested vote from peers 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937), 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983)
I20240501 08:13:43.233973 7007 raft_consensus.cc:686] T 19ec6a8b857d428d858fa831e825c578 P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Becoming Leader. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:43.234524 7405 raft_consensus.cc:2781] T 416ef8d814614a7caba4279e64893b27 P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:43.235718 5441 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4c29ea3b807d4da1b970246c64797a91"
dest_uuid: "2c4d718738d1408eabce489f4e140af6"
from {username='slave'} at 127.0.0.1:37632
I20240501 08:13:43.236048 7007 consensus_queue.cc:237] T 19ec6a8b857d428d858fa831e825c578 P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.236512 7405 raft_consensus.cc:686] T 416ef8d814614a7caba4279e64893b27 P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Becoming Leader. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
W20240501 08:13:43.237293 5377 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.84.131:36983, user_credentials={real_user=slave}} blocked reactor thread for 74011.5us
I20240501 08:13:43.236646 5441 raft_consensus.cc:465] T 4c29ea3b807d4da1b970246c64797a91 P 2c4d718738d1408eabce489f4e140af6 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 08:13:43.238751 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14d9bd97e4674e0ebdd90bf45b5679cb" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
I20240501 08:13:43.239001 5311 raft_consensus.cc:3037] T 14d9bd97e4674e0ebdd90bf45b5679cb P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.238852 5440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1835b9a1f198420c82e1994a3056ea9a"
dest_uuid: "2c4d718738d1408eabce489f4e140af6"
from {username='slave'} at 127.0.0.1:37632
I20240501 08:13:43.239284 5440 raft_consensus.cc:482] T 1835b9a1f198420c82e1994a3056ea9a P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.239480 5440 raft_consensus.cc:3037] T 1835b9a1f198420c82e1994a3056ea9a P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.241766 5311 raft_consensus.cc:2454] T 14d9bd97e4674e0ebdd90bf45b5679cb P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 2.
I20240501 08:13:43.242870 5050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14d9bd97e4674e0ebdd90bf45b5679cb" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:43.243079 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "416ef8d814614a7caba4279e64893b27" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:43.243288 5051 raft_consensus.cc:3037] T 416ef8d814614a7caba4279e64893b27 P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.243314 5050 raft_consensus.cc:3032] T 14d9bd97e4674e0ebdd90bf45b5679cb P b6cc443a494e4595b0e7cf1777620a3f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:43.243505 5050 raft_consensus.cc:729] T 14d9bd97e4674e0ebdd90bf45b5679cb P b6cc443a494e4595b0e7cf1777620a3f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b6cc443a494e4595b0e7cf1777620a3f, State: Running, Role: LEADER
I20240501 08:13:43.244060 5050 consensus_queue.cc:260] T 14d9bd97e4674e0ebdd90bf45b5679cb P b6cc443a494e4595b0e7cf1777620a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.244793 5050 raft_consensus.cc:3037] T 14d9bd97e4674e0ebdd90bf45b5679cb P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.246086 5051 raft_consensus.cc:2454] T 416ef8d814614a7caba4279e64893b27 P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 2.
I20240501 08:13:43.247123 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e30e05c0ef498288f35706978c981b" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:43.247347 5051 raft_consensus.cc:3037] T e9e30e05c0ef498288f35706978c981b P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.248622 5050 raft_consensus.cc:2454] T 14d9bd97e4674e0ebdd90bf45b5679cb P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 2.
I20240501 08:13:43.249089 7405 consensus_queue.cc:237] T 416ef8d814614a7caba4279e64893b27 P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
W20240501 08:13:43.250180 5376 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.84.130:41937, user_credentials={real_user=slave}} blocked reactor thread for 43909.7us
I20240501 08:13:43.250277 5051 raft_consensus.cc:2454] T e9e30e05c0ef498288f35706978c981b P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 2.
I20240501 08:13:43.251390 5181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e30e05c0ef498288f35706978c981b" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:43.251662 5181 raft_consensus.cc:3032] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:43.251791 5181 raft_consensus.cc:729] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Running, Role: LEADER
I20240501 08:13:43.252051 5181 consensus_queue.cc:260] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.252450 5181 raft_consensus.cc:3037] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.254127 5440 raft_consensus.cc:504] T 1835b9a1f198420c82e1994a3056ea9a P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:43.254757 5440 leader_election.cc:288] T 1835b9a1f198420c82e1994a3056ea9a P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 2 election: Requested vote from peers 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937), b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417)
I20240501 08:13:43.255048 5181 raft_consensus.cc:2454] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 2.
I20240501 08:13:43.255162 5182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1835b9a1f198420c82e1994a3056ea9a" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:43.255403 5182 raft_consensus.cc:3037] T 1835b9a1f198420c82e1994a3056ea9a P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.255412 5180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff3b86014f2940b18707803c38bb5b8f" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:43.255820 5441 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0ad7f7e55fd34688b5eba16555306bc9"
dest_uuid: "2c4d718738d1408eabce489f4e140af6"
from {username='slave'} at 127.0.0.1:37632
I20240501 08:13:43.255990 5180 raft_consensus.cc:3037] T ff3b86014f2940b18707803c38bb5b8f P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.256024 5441 raft_consensus.cc:482] T 0ad7f7e55fd34688b5eba16555306bc9 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.256124 5441 raft_consensus.cc:3037] T 0ad7f7e55fd34688b5eba16555306bc9 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.258160 5182 raft_consensus.cc:2454] T 1835b9a1f198420c82e1994a3056ea9a P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 2.
I20240501 08:13:43.258546 5180 raft_consensus.cc:2454] T ff3b86014f2940b18707803c38bb5b8f P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 2.
I20240501 08:13:43.258636 5441 raft_consensus.cc:504] T 0ad7f7e55fd34688b5eba16555306bc9 P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:43.259184 5376 leader_election.cc:302] T e9e30e05c0ef498288f35706978c981b P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 2c4d718738d1408eabce489f4e140af6; no voters:
I20240501 08:13:43.259299 5441 leader_election.cc:288] T 0ad7f7e55fd34688b5eba16555306bc9 P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 2 election: Requested vote from peers 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983), b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417)
I20240501 08:13:43.259580 5376 leader_election.cc:302] T 1835b9a1f198420c82e1994a3056ea9a P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 2c4d718738d1408eabce489f4e140af6; no voters:
I20240501 08:13:43.259791 7404 raft_consensus.cc:2781] T e9e30e05c0ef498288f35706978c981b P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:43.260100 7407 raft_consensus.cc:2781] T 1835b9a1f198420c82e1994a3056ea9a P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:43.260252 7407 raft_consensus.cc:686] T 1835b9a1f198420c82e1994a3056ea9a P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Becoming Leader. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:43.260288 5439 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "98e5387cc69b483796e42cbf54f2226d"
dest_uuid: "2c4d718738d1408eabce489f4e140af6"
from {username='slave'} at 127.0.0.1:37632
I20240501 08:13:43.260224 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff3b86014f2940b18707803c38bb5b8f" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
I20240501 08:13:43.260452 5439 raft_consensus.cc:482] T 98e5387cc69b483796e42cbf54f2226d P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.260452 5311 raft_consensus.cc:3032] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:43.260571 5439 raft_consensus.cc:3037] T 98e5387cc69b483796e42cbf54f2226d P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.260566 5311 raft_consensus.cc:729] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:43.260535 7407 consensus_queue.cc:237] T 1835b9a1f198420c82e1994a3056ea9a P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:43.260983 5180 raft_consensus.cc:1226] T 1835b9a1f198420c82e1994a3056ea9a P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Rejecting Update request from peer b6cc443a494e4595b0e7cf1777620a3f for earlier term 1. Current term is 2. Ops: []
I20240501 08:13:43.261575 6631 consensus_queue.cc:1046] T 1835b9a1f198420c82e1994a3056ea9a P b6cc443a494e4595b0e7cf1777620a3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:43.262073 5311 consensus_queue.cc:260] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.262416 7404 raft_consensus.cc:686] T e9e30e05c0ef498288f35706978c981b P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Becoming Leader. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:43.262490 5311 raft_consensus.cc:3037] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.262842 7404 consensus_queue.cc:237] T e9e30e05c0ef498288f35706978c981b P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.263584 5439 raft_consensus.cc:504] T 98e5387cc69b483796e42cbf54f2226d P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:43.264238 5439 leader_election.cc:288] T 98e5387cc69b483796e42cbf54f2226d P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 2 election: Requested vote from peers 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937), b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417)
I20240501 08:13:43.265381 5442 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e13f0b2e652c44fc994e9f950d40becb"
dest_uuid: "2c4d718738d1408eabce489f4e140af6"
from {username='slave'} at 127.0.0.1:37632
I20240501 08:13:43.265575 5442 raft_consensus.cc:465] T e13f0b2e652c44fc994e9f950d40becb P 2c4d718738d1408eabce489f4e140af6 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 08:13:43.266053 5376 leader_election.cc:302] T ff3b86014f2940b18707803c38bb5b8f P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 2c4d718738d1408eabce489f4e140af6; no voters:
I20240501 08:13:43.266321 7407 raft_consensus.cc:2781] T ff3b86014f2940b18707803c38bb5b8f P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:43.266517 7407 raft_consensus.cc:686] T ff3b86014f2940b18707803c38bb5b8f P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Becoming Leader. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:43.266773 7407 consensus_queue.cc:237] T ff3b86014f2940b18707803c38bb5b8f P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.268388 5442 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f518dc436d1d4e9dbe0ec9bbaf17e5fd"
dest_uuid: "2c4d718738d1408eabce489f4e140af6"
from {username='slave'} at 127.0.0.1:37632
I20240501 08:13:43.268597 5442 raft_consensus.cc:482] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.268690 5442 raft_consensus.cc:3037] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.271454 5442 raft_consensus.cc:504] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:43.272109 5442 leader_election.cc:288] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 2 election: Requested vote from peers 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983), 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937)
I20240501 08:13:43.272616 5180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e5387cc69b483796e42cbf54f2226d" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:43.272955 5180 raft_consensus.cc:3032] T 98e5387cc69b483796e42cbf54f2226d P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:43.273130 5180 raft_consensus.cc:729] T 98e5387cc69b483796e42cbf54f2226d P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Running, Role: LEADER
I20240501 08:13:43.273571 5180 consensus_queue.cc:260] T 98e5387cc69b483796e42cbf54f2226d P 0a952997a9fc4bfb8d7cbc56de6e0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:43.274101 5180 raft_consensus.cc:3037] T 98e5387cc69b483796e42cbf54f2226d P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.274247 5442 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f8f72bb32e34420eab300223e4428591"
dest_uuid: "2c4d718738d1408eabce489f4e140af6"
from {username='slave'} at 127.0.0.1:37632
I20240501 08:13:43.274531 5442 raft_consensus.cc:482] T f8f72bb32e34420eab300223e4428591 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.274745 5442 raft_consensus.cc:3037] T f8f72bb32e34420eab300223e4428591 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.277029 5180 raft_consensus.cc:2454] T 98e5387cc69b483796e42cbf54f2226d P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 2.
I20240501 08:13:43.277850 5311 raft_consensus.cc:2454] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 2.
I20240501 08:13:43.278550 5180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f518dc436d1d4e9dbe0ec9bbaf17e5fd" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:43.278735 5180 raft_consensus.cc:3032] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:43.278826 5180 raft_consensus.cc:729] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Running, Role: LEADER
I20240501 08:13:43.279064 5180 consensus_queue.cc:260] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 0a952997a9fc4bfb8d7cbc56de6e0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:43.279414 5180 raft_consensus.cc:3037] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.282724 5442 raft_consensus.cc:504] T f8f72bb32e34420eab300223e4428591 P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:43.284008 5442 leader_election.cc:288] T f8f72bb32e34420eab300223e4428591 P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 2 election: Requested vote from peers 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983), 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937)
I20240501 08:13:43.284216 5376 leader_election.cc:302] T 98e5387cc69b483796e42cbf54f2226d P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 2c4d718738d1408eabce489f4e140af6; no voters:
I20240501 08:13:43.284606 4909 catalog_manager.cc:5484] T 3d668173d46d48ff840a535273009626 P 2c4d718738d1408eabce489f4e140af6 reported cstate change: term changed from 1 to 2, leader changed from 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131) to 2c4d718738d1408eabce489f4e140af6 (127.21.84.132). New cstate: current_term: 2 leader_uuid: "2c4d718738d1408eabce489f4e140af6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:43.285333 5181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8f72bb32e34420eab300223e4428591" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:43.285537 5181 raft_consensus.cc:3032] T f8f72bb32e34420eab300223e4428591 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:43.285627 5181 raft_consensus.cc:729] T f8f72bb32e34420eab300223e4428591 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Running, Role: LEADER
I20240501 08:13:43.285403 4909 catalog_manager.cc:5484] T 416ef8d814614a7caba4279e64893b27 P 2c4d718738d1408eabce489f4e140af6 reported cstate change: term changed from 1 to 2, leader changed from 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130) to 2c4d718738d1408eabce489f4e140af6 (127.21.84.132). New cstate: current_term: 2 leader_uuid: "2c4d718738d1408eabce489f4e140af6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:43.285910 5181 consensus_queue.cc:260] T f8f72bb32e34420eab300223e4428591 P 0a952997a9fc4bfb8d7cbc56de6e0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:43.286324 5181 raft_consensus.cc:3037] T f8f72bb32e34420eab300223e4428591 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.288017 4909 catalog_manager.cc:5484] T 86ffaa5522df469e9bb410872fae256d P 2c4d718738d1408eabce489f4e140af6 reported cstate change: term changed from 1 to 2, leader changed from b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129) to 2c4d718738d1408eabce489f4e140af6 (127.21.84.132). New cstate: current_term: 2 leader_uuid: "2c4d718738d1408eabce489f4e140af6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:43.288857 4909 catalog_manager.cc:5484] T 2e0d1fd6767b454e9871e1bdec86caa8 P 2c4d718738d1408eabce489f4e140af6 reported cstate change: term changed from 1 to 2, leader changed from 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130) to 2c4d718738d1408eabce489f4e140af6 (127.21.84.132). New cstate: current_term: 2 leader_uuid: "2c4d718738d1408eabce489f4e140af6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: HEALTHY } } }
I20240501 08:13:43.289602 5181 raft_consensus.cc:2454] T f8f72bb32e34420eab300223e4428591 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 2.
I20240501 08:13:43.290215 7407 raft_consensus.cc:2781] T 98e5387cc69b483796e42cbf54f2226d P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:43.289614 4909 catalog_manager.cc:5484] T 1835b9a1f198420c82e1994a3056ea9a P 2c4d718738d1408eabce489f4e140af6 reported cstate change: term changed from 1 to 2, leader changed from b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129) to 2c4d718738d1408eabce489f4e140af6 (127.21.84.132). New cstate: current_term: 2 leader_uuid: "2c4d718738d1408eabce489f4e140af6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:43.290968 7407 raft_consensus.cc:686] T 98e5387cc69b483796e42cbf54f2226d P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Becoming Leader. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:43.290910 4909 catalog_manager.cc:5484] T e9e30e05c0ef498288f35706978c981b P 2c4d718738d1408eabce489f4e140af6 reported cstate change: term changed from 1 to 2, leader changed from 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130) to 2c4d718738d1408eabce489f4e140af6 (127.21.84.132). New cstate: current_term: 2 leader_uuid: "2c4d718738d1408eabce489f4e140af6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: HEALTHY } } }
I20240501 08:13:43.291258 7407 consensus_queue.cc:237] T 98e5387cc69b483796e42cbf54f2226d P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:43.292001 4909 catalog_manager.cc:5484] T 19ec6a8b857d428d858fa831e825c578 P 2c4d718738d1408eabce489f4e140af6 reported cstate change: term changed from 1 to 2, leader changed from 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131) to 2c4d718738d1408eabce489f4e140af6 (127.21.84.132). New cstate: current_term: 2 leader_uuid: "2c4d718738d1408eabce489f4e140af6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: HEALTHY } } }
I20240501 08:13:43.296173 5376 leader_election.cc:302] T f8f72bb32e34420eab300223e4428591 P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 2c4d718738d1408eabce489f4e140af6; no voters:
I20240501 08:13:43.296993 7007 raft_consensus.cc:2781] T f8f72bb32e34420eab300223e4428591 P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:43.297145 7007 raft_consensus.cc:686] T f8f72bb32e34420eab300223e4428591 P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Becoming Leader. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:43.297403 7007 consensus_queue.cc:237] T f8f72bb32e34420eab300223e4428591 P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:43.301362 5442 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "97943d9ec11f4c5da7a33a91818d9978"
dest_uuid: "2c4d718738d1408eabce489f4e140af6"
from {username='slave'} at 127.0.0.1:37632
I20240501 08:13:43.301641 5442 raft_consensus.cc:482] T 97943d9ec11f4c5da7a33a91818d9978 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.301793 5442 raft_consensus.cc:3037] T 97943d9ec11f4c5da7a33a91818d9978 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.303156 5377 leader_election.cc:302] T a3445b99b36444bb811486be77100479 P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4d718738d1408eabce489f4e140af6, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:43.303465 7407 raft_consensus.cc:2781] T a3445b99b36444bb811486be77100479 P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:43.303640 7407 raft_consensus.cc:686] T a3445b99b36444bb811486be77100479 P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Becoming Leader. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:43.303920 7407 consensus_queue.cc:237] T a3445b99b36444bb811486be77100479 P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:43.305250 5377 leader_election.cc:302] T 14d9bd97e4674e0ebdd90bf45b5679cb P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4d718738d1408eabce489f4e140af6, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:43.306496 4909 catalog_manager.cc:5484] T ff3b86014f2940b18707803c38bb5b8f P 2c4d718738d1408eabce489f4e140af6 reported cstate change: term changed from 1 to 2, leader changed from 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131) to 2c4d718738d1408eabce489f4e140af6 (127.21.84.132). New cstate: current_term: 2 leader_uuid: "2c4d718738d1408eabce489f4e140af6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: HEALTHY } } }
I20240501 08:13:43.307045 5180 raft_consensus.cc:2454] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 2.
I20240501 08:13:43.307538 7007 raft_consensus.cc:2781] T 14d9bd97e4674e0ebdd90bf45b5679cb P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:43.307772 7007 raft_consensus.cc:686] T 14d9bd97e4674e0ebdd90bf45b5679cb P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Becoming Leader. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:43.308094 7007 consensus_queue.cc:237] T 14d9bd97e4674e0ebdd90bf45b5679cb P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.308566 5376 leader_election.cc:302] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 2c4d718738d1408eabce489f4e140af6; no voters:
I20240501 08:13:43.309019 7404 raft_consensus.cc:2781] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:43.309324 7404 raft_consensus.cc:686] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Becoming Leader. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:43.309760 7404 consensus_queue.cc:237] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:43.311275 4909 catalog_manager.cc:5484] T f8f72bb32e34420eab300223e4428591 P 2c4d718738d1408eabce489f4e140af6 reported cstate change: term changed from 1 to 2, leader changed from 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130) to 2c4d718738d1408eabce489f4e140af6 (127.21.84.132). New cstate: current_term: 2 leader_uuid: "2c4d718738d1408eabce489f4e140af6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:43.312211 4909 catalog_manager.cc:5484] T 98e5387cc69b483796e42cbf54f2226d P 2c4d718738d1408eabce489f4e140af6 reported cstate change: term changed from 1 to 2, leader changed from 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130) to 2c4d718738d1408eabce489f4e140af6 (127.21.84.132). New cstate: current_term: 2 leader_uuid: "2c4d718738d1408eabce489f4e140af6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:43.314116 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1835b9a1f198420c82e1994a3056ea9a" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:43.315363 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ad7f7e55fd34688b5eba16555306bc9" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
I20240501 08:13:43.315650 5311 raft_consensus.cc:3032] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:43.315809 5311 raft_consensus.cc:729] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:43.316156 5311 consensus_queue.cc:260] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:43.316414 5050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ad7f7e55fd34688b5eba16555306bc9" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:43.316655 5050 raft_consensus.cc:3037] T 0ad7f7e55fd34688b5eba16555306bc9 P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.316689 5311 raft_consensus.cc:3037] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.319569 5050 raft_consensus.cc:2454] T 0ad7f7e55fd34688b5eba16555306bc9 P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 2.
I20240501 08:13:43.320633 5050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98e5387cc69b483796e42cbf54f2226d" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:43.320937 5311 raft_consensus.cc:2454] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 2.
I20240501 08:13:43.321133 5050 raft_consensus.cc:3037] T 98e5387cc69b483796e42cbf54f2226d P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.324179 5050 raft_consensus.cc:2454] T 98e5387cc69b483796e42cbf54f2226d P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 2.
I20240501 08:13:43.325291 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f518dc436d1d4e9dbe0ec9bbaf17e5fd" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
I20240501 08:13:43.325564 5311 raft_consensus.cc:3037] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.326069 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8f72bb32e34420eab300223e4428591" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
I20240501 08:13:43.326328 5309 raft_consensus.cc:3037] T f8f72bb32e34420eab300223e4428591 P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.326638 5377 leader_election.cc:302] T 0ad7f7e55fd34688b5eba16555306bc9 P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4d718738d1408eabce489f4e140af6, 9808124b81e14d1290ab0a6641a4b836; no voters:
I20240501 08:13:43.327442 5051 raft_consensus.cc:3032] T 1835b9a1f198420c82e1994a3056ea9a P b6cc443a494e4595b0e7cf1777620a3f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:43.327550 5051 raft_consensus.cc:729] T 1835b9a1f198420c82e1994a3056ea9a P b6cc443a494e4595b0e7cf1777620a3f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b6cc443a494e4595b0e7cf1777620a3f, State: Running, Role: LEADER
I20240501 08:13:43.327807 5051 consensus_queue.cc:260] T 1835b9a1f198420c82e1994a3056ea9a P b6cc443a494e4595b0e7cf1777620a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:43.328007 7007 raft_consensus.cc:2781] T 0ad7f7e55fd34688b5eba16555306bc9 P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:43.328147 5051 raft_consensus.cc:3037] T 1835b9a1f198420c82e1994a3056ea9a P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.328188 7007 raft_consensus.cc:686] T 0ad7f7e55fd34688b5eba16555306bc9 P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Becoming Leader. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:43.328475 7007 consensus_queue.cc:237] T 0ad7f7e55fd34688b5eba16555306bc9 P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:43.329303 5309 raft_consensus.cc:2454] T f8f72bb32e34420eab300223e4428591 P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 2.
I20240501 08:13:43.330729 5442 raft_consensus.cc:504] T 97943d9ec11f4c5da7a33a91818d9978 P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.330910 5051 raft_consensus.cc:2454] T 1835b9a1f198420c82e1994a3056ea9a P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 2.
W20240501 08:13:43.331717 7421 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20240501 08:13:43.332350 5311 raft_consensus.cc:2454] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 2.
I20240501 08:13:43.335219 5442 leader_election.cc:288] T 97943d9ec11f4c5da7a33a91818d9978 P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 2 election: Requested vote from peers 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937), 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983)
I20240501 08:13:43.335616 5180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97943d9ec11f4c5da7a33a91818d9978" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:43.335903 5180 raft_consensus.cc:3032] T 97943d9ec11f4c5da7a33a91818d9978 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:43.336050 5180 raft_consensus.cc:729] T 97943d9ec11f4c5da7a33a91818d9978 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Running, Role: LEADER
I20240501 08:13:43.336373 5180 consensus_queue.cc:260] T 97943d9ec11f4c5da7a33a91818d9978 P 0a952997a9fc4bfb8d7cbc56de6e0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.336865 5180 raft_consensus.cc:3037] T 97943d9ec11f4c5da7a33a91818d9978 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.339749 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97943d9ec11f4c5da7a33a91818d9978" candidate_uuid: "2c4d718738d1408eabce489f4e140af6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
I20240501 08:13:43.340003 5311 raft_consensus.cc:3037] T 97943d9ec11f4c5da7a33a91818d9978 P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.341449 5180 raft_consensus.cc:2454] T 97943d9ec11f4c5da7a33a91818d9978 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 2.
I20240501 08:13:43.342069 5376 leader_election.cc:302] T 97943d9ec11f4c5da7a33a91818d9978 P 2c4d718738d1408eabce489f4e140af6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 2c4d718738d1408eabce489f4e140af6; no voters:
I20240501 08:13:43.342413 7007 raft_consensus.cc:2781] T 97943d9ec11f4c5da7a33a91818d9978 P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:43.342567 7007 raft_consensus.cc:686] T 97943d9ec11f4c5da7a33a91818d9978 P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Becoming Leader. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:43.342790 5311 raft_consensus.cc:2454] T 97943d9ec11f4c5da7a33a91818d9978 P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4d718738d1408eabce489f4e140af6 in term 2.
I20240501 08:13:43.342823 7007 consensus_queue.cc:237] T 97943d9ec11f4c5da7a33a91818d9978 P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.344633 4909 catalog_manager.cc:5484] T a3445b99b36444bb811486be77100479 P 2c4d718738d1408eabce489f4e140af6 reported cstate change: term changed from 1 to 2, leader changed from b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129) to 2c4d718738d1408eabce489f4e140af6 (127.21.84.132). New cstate: current_term: 2 leader_uuid: "2c4d718738d1408eabce489f4e140af6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:43.345423 4909 catalog_manager.cc:5484] T 0ad7f7e55fd34688b5eba16555306bc9 P 2c4d718738d1408eabce489f4e140af6 reported cstate change: term changed from 1 to 2, leader changed from 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131) to 2c4d718738d1408eabce489f4e140af6 (127.21.84.132). New cstate: current_term: 2 leader_uuid: "2c4d718738d1408eabce489f4e140af6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:43.346040 4909 catalog_manager.cc:5484] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 2c4d718738d1408eabce489f4e140af6 reported cstate change: term changed from 1 to 2, leader changed from 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130) to 2c4d718738d1408eabce489f4e140af6 (127.21.84.132). New cstate: current_term: 2 leader_uuid: "2c4d718738d1408eabce489f4e140af6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:43.346740 4909 catalog_manager.cc:5484] T 14d9bd97e4674e0ebdd90bf45b5679cb P 2c4d718738d1408eabce489f4e140af6 reported cstate change: term changed from 1 to 2, leader changed from b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129) to 2c4d718738d1408eabce489f4e140af6 (127.21.84.132). New cstate: current_term: 2 leader_uuid: "2c4d718738d1408eabce489f4e140af6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: HEALTHY } } }
I20240501 08:13:43.355158 4909 catalog_manager.cc:5484] T 97943d9ec11f4c5da7a33a91818d9978 P 2c4d718738d1408eabce489f4e140af6 reported cstate change: term changed from 1 to 2, leader changed from 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130) to 2c4d718738d1408eabce489f4e140af6 (127.21.84.132). New cstate: current_term: 2 leader_uuid: "2c4d718738d1408eabce489f4e140af6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: HEALTHY } } }
I20240501 08:13:43.391618 5311 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5e1018f0fee24311b58936ace2f6d53b"
dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
from {username='slave'} at 127.0.0.1:57234
I20240501 08:13:43.391968 5311 raft_consensus.cc:482] T 5e1018f0fee24311b58936ace2f6d53b P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.392151 5311 raft_consensus.cc:3037] T 5e1018f0fee24311b58936ace2f6d53b P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.395211 5311 raft_consensus.cc:504] T 5e1018f0fee24311b58936ace2f6d53b P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:43.396492 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1018f0fee24311b58936ace2f6d53b" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:43.396592 5180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1018f0fee24311b58936ace2f6d53b" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:43.396715 5051 raft_consensus.cc:3037] T 5e1018f0fee24311b58936ace2f6d53b P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.396809 5180 raft_consensus.cc:3032] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:43.396901 5180 raft_consensus.cc:729] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Running, Role: LEADER
I20240501 08:13:43.397183 5180 consensus_queue.cc:260] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:43.397564 5180 raft_consensus.cc:3037] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.399564 5051 raft_consensus.cc:2454] T 5e1018f0fee24311b58936ace2f6d53b P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 2.
I20240501 08:13:43.400277 5180 raft_consensus.cc:2454] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 2.
I20240501 08:13:43.400265 5247 leader_election.cc:302] T 5e1018f0fee24311b58936ace2f6d53b P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9808124b81e14d1290ab0a6641a4b836, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:43.400635 7139 raft_consensus.cc:2781] T 5e1018f0fee24311b58936ace2f6d53b P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:43.401062 5311 leader_election.cc:288] T 5e1018f0fee24311b58936ace2f6d53b P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 2 election: Requested vote from peers b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417), 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937)
I20240501 08:13:43.404187 5309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "19585175235c4877b9c4da931c345565"
dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
from {username='slave'} at 127.0.0.1:57234
I20240501 08:13:43.404424 5309 raft_consensus.cc:482] T 19585175235c4877b9c4da931c345565 P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.404590 5309 raft_consensus.cc:3037] T 19585175235c4877b9c4da931c345565 P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.408488 5309 raft_consensus.cc:504] T 19585175235c4877b9c4da931c345565 P 9808124b81e14d1290ab0a6641a4b836 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.409198 7139 raft_consensus.cc:686] T 5e1018f0fee24311b58936ace2f6d53b P 9808124b81e14d1290ab0a6641a4b836 [term 2 LEADER]: Becoming Leader. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:43.409880 5180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19585175235c4877b9c4da931c345565" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:43.410173 5180 raft_consensus.cc:3037] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.409634 7139 consensus_queue.cc:237] T 5e1018f0fee24311b58936ace2f6d53b P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:43.410223 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19585175235c4877b9c4da931c345565" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:43.410436 5442 raft_consensus.cc:3032] T 19585175235c4877b9c4da931c345565 P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:13:43.410562 5442 raft_consensus.cc:729] T 19585175235c4877b9c4da931c345565 P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:43.410872 5442 consensus_queue.cc:260] T 19585175235c4877b9c4da931c345565 P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.411372 5442 raft_consensus.cc:3037] T 19585175235c4877b9c4da931c345565 P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.413008 5180 raft_consensus.cc:2454] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 3.
I20240501 08:13:43.413534 5246 leader_election.cc:302] T 19585175235c4877b9c4da931c345565 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 9808124b81e14d1290ab0a6641a4b836; no voters:
I20240501 08:13:43.414021 7434 raft_consensus.cc:2781] T 19585175235c4877b9c4da931c345565 P 9808124b81e14d1290ab0a6641a4b836 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:13:43.415180 5442 raft_consensus.cc:2396] T 19585175235c4877b9c4da931c345565 P 2c4d718738d1408eabce489f4e140af6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9808124b81e14d1290ab0a6641a4b836 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:13:43.416252 5309 leader_election.cc:288] T 19585175235c4877b9c4da931c345565 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 3 election: Requested vote from peers 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937), 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797)
I20240501 08:13:43.417629 7434 raft_consensus.cc:686] T 19585175235c4877b9c4da931c345565 P 9808124b81e14d1290ab0a6641a4b836 [term 3 LEADER]: Becoming Leader. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:43.417682 5309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "faf43fec190c4830957d4e3ab304ba9b"
dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
from {username='slave'} at 127.0.0.1:57234
I20240501 08:13:43.417979 5309 raft_consensus.cc:482] T faf43fec190c4830957d4e3ab304ba9b P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.418129 5309 raft_consensus.cc:3037] T faf43fec190c4830957d4e3ab304ba9b P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.418141 7434 consensus_queue.cc:237] T 19585175235c4877b9c4da931c345565 P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.420837 5309 raft_consensus.cc:504] T faf43fec190c4830957d4e3ab304ba9b P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:43.421840 5050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf43fec190c4830957d4e3ab304ba9b" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:43.421916 5180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf43fec190c4830957d4e3ab304ba9b" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:43.421667 4909 catalog_manager.cc:5484] T 5e1018f0fee24311b58936ace2f6d53b P 9808124b81e14d1290ab0a6641a4b836 reported cstate change: term changed from 1 to 2, leader changed from 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130) to 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131). New cstate: current_term: 2 leader_uuid: "9808124b81e14d1290ab0a6641a4b836" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:43.422158 5050 raft_consensus.cc:3037] T faf43fec190c4830957d4e3ab304ba9b P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.422158 5180 raft_consensus.cc:3032] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:43.422246 5180 raft_consensus.cc:729] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Running, Role: LEADER
I20240501 08:13:43.422889 5309 leader_election.cc:288] T faf43fec190c4830957d4e3ab304ba9b P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 2 election: Requested vote from peers b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417), 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937)
I20240501 08:13:43.423311 5180 consensus_queue.cc:260] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:43.423662 5180 raft_consensus.cc:3037] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.424984 5050 raft_consensus.cc:2454] T faf43fec190c4830957d4e3ab304ba9b P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 2.
I20240501 08:13:43.428464 5247 leader_election.cc:302] T faf43fec190c4830957d4e3ab304ba9b P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9808124b81e14d1290ab0a6641a4b836, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:43.430140 5309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ab68600377eb4581a724b714cfe234d7"
dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
from {username='slave'} at 127.0.0.1:57234
I20240501 08:13:43.430272 5180 raft_consensus.cc:2454] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 2.
I20240501 08:13:43.430438 5309 raft_consensus.cc:482] T ab68600377eb4581a724b714cfe234d7 P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.430621 5309 raft_consensus.cc:3037] T ab68600377eb4581a724b714cfe234d7 P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.434036 7436 raft_consensus.cc:2781] T faf43fec190c4830957d4e3ab304ba9b P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:43.434499 5309 raft_consensus.cc:504] T ab68600377eb4581a724b714cfe234d7 P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:43.436106 5180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab68600377eb4581a724b714cfe234d7" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:43.436448 5180 raft_consensus.cc:3037] T ab68600377eb4581a724b714cfe234d7 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.438050 5309 leader_election.cc:288] T ab68600377eb4581a724b714cfe234d7 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 2 election: Requested vote from peers 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797), 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937)
I20240501 08:13:43.434633 7436 raft_consensus.cc:686] T faf43fec190c4830957d4e3ab304ba9b P 9808124b81e14d1290ab0a6641a4b836 [term 2 LEADER]: Becoming Leader. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:43.439515 5311 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c1bd10f523444d668736a2ff5e02442c"
dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
from {username='slave'} at 127.0.0.1:57234
I20240501 08:13:43.439736 5311 raft_consensus.cc:482] T c1bd10f523444d668736a2ff5e02442c P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.439857 5311 raft_consensus.cc:3037] T c1bd10f523444d668736a2ff5e02442c P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.441344 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab68600377eb4581a724b714cfe234d7" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:43.441588 5442 raft_consensus.cc:3032] T ab68600377eb4581a724b714cfe234d7 P 2c4d718738d1408eabce489f4e140af6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:43.441697 5442 raft_consensus.cc:729] T ab68600377eb4581a724b714cfe234d7 P 2c4d718738d1408eabce489f4e140af6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:43.441706 5180 raft_consensus.cc:2454] T ab68600377eb4581a724b714cfe234d7 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 2.
I20240501 08:13:43.442009 5442 consensus_queue.cc:260] T ab68600377eb4581a724b714cfe234d7 P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:43.442189 5246 leader_election.cc:302] T ab68600377eb4581a724b714cfe234d7 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 9808124b81e14d1290ab0a6641a4b836; no voters:
I20240501 08:13:43.442411 5442 raft_consensus.cc:3037] T ab68600377eb4581a724b714cfe234d7 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.442598 7436 consensus_queue.cc:237] T faf43fec190c4830957d4e3ab304ba9b P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:43.442984 5311 raft_consensus.cc:504] T c1bd10f523444d668736a2ff5e02442c P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.444082 5181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1bd10f523444d668736a2ff5e02442c" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:43.444288 5181 raft_consensus.cc:3037] T c1bd10f523444d668736a2ff5e02442c P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.447118 5181 raft_consensus.cc:2454] T c1bd10f523444d668736a2ff5e02442c P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 2.
I20240501 08:13:43.447595 5246 leader_election.cc:302] T c1bd10f523444d668736a2ff5e02442c P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 9808124b81e14d1290ab0a6641a4b836; no voters:
I20240501 08:13:43.447978 5311 leader_election.cc:288] T c1bd10f523444d668736a2ff5e02442c P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 2 election: Requested vote from peers 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937), 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797)
I20240501 08:13:43.448954 5309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "14d9bd97e4674e0ebdd90bf45b5679cb"
dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
from {username='slave'} at 127.0.0.1:57234
I20240501 08:13:43.449148 5309 raft_consensus.cc:482] T 14d9bd97e4674e0ebdd90bf45b5679cb P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.449254 5309 raft_consensus.cc:3037] T 14d9bd97e4674e0ebdd90bf45b5679cb P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.451717 7139 raft_consensus.cc:2781] T ab68600377eb4581a724b714cfe234d7 P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:43.452821 5442 raft_consensus.cc:2454] T ab68600377eb4581a724b714cfe234d7 P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 2.
I20240501 08:13:43.453204 5309 raft_consensus.cc:504] T 14d9bd97e4674e0ebdd90bf45b5679cb P 9808124b81e14d1290ab0a6641a4b836 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.453446 7436 raft_consensus.cc:2781] T c1bd10f523444d668736a2ff5e02442c P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:43.453676 7436 raft_consensus.cc:686] T c1bd10f523444d668736a2ff5e02442c P 9808124b81e14d1290ab0a6641a4b836 [term 2 LEADER]: Becoming Leader. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:43.454087 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1bd10f523444d668736a2ff5e02442c" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:43.454186 5309 leader_election.cc:288] T 14d9bd97e4674e0ebdd90bf45b5679cb P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 3 election: Requested vote from peers b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417), 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797)
I20240501 08:13:43.454303 5442 raft_consensus.cc:3032] T c1bd10f523444d668736a2ff5e02442c P 2c4d718738d1408eabce489f4e140af6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:43.454406 5442 raft_consensus.cc:729] T c1bd10f523444d668736a2ff5e02442c P 2c4d718738d1408eabce489f4e140af6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:43.454450 5050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14d9bd97e4674e0ebdd90bf45b5679cb" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:43.454646 5050 raft_consensus.cc:3037] T 14d9bd97e4674e0ebdd90bf45b5679cb P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.454737 5442 consensus_queue.cc:260] T c1bd10f523444d668736a2ff5e02442c P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.455143 5311 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a3445b99b36444bb811486be77100479"
dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
from {username='slave'} at 127.0.0.1:57234
I20240501 08:13:43.455363 5311 raft_consensus.cc:482] T a3445b99b36444bb811486be77100479 P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.455461 5311 raft_consensus.cc:3037] T a3445b99b36444bb811486be77100479 P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.456822 5442 raft_consensus.cc:3037] T c1bd10f523444d668736a2ff5e02442c P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.457306 5050 raft_consensus.cc:2454] T 14d9bd97e4674e0ebdd90bf45b5679cb P b6cc443a494e4595b0e7cf1777620a3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 3.
I20240501 08:13:43.458312 5247 leader_election.cc:302] T 14d9bd97e4674e0ebdd90bf45b5679cb P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9808124b81e14d1290ab0a6641a4b836, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:43.458750 5311 raft_consensus.cc:504] T a3445b99b36444bb811486be77100479 P 9808124b81e14d1290ab0a6641a4b836 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:43.459481 5311 leader_election.cc:288] T a3445b99b36444bb811486be77100479 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 3 election: Requested vote from peers 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797), b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417)
I20240501 08:13:43.459496 5442 raft_consensus.cc:2454] T c1bd10f523444d668736a2ff5e02442c P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 2.
I20240501 08:13:43.463449 7436 consensus_queue.cc:237] T c1bd10f523444d668736a2ff5e02442c P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.464778 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14d9bd97e4674e0ebdd90bf45b5679cb" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:43.465075 5442 raft_consensus.cc:3032] T 14d9bd97e4674e0ebdd90bf45b5679cb P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:13:43.465363 5442 raft_consensus.cc:729] T 14d9bd97e4674e0ebdd90bf45b5679cb P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:43.466063 5440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3445b99b36444bb811486be77100479" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:43.466632 5440 raft_consensus.cc:3032] T a3445b99b36444bb811486be77100479 P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:13:43.466825 5440 raft_consensus.cc:729] T a3445b99b36444bb811486be77100479 P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:43.467530 5440 consensus_queue.cc:260] T a3445b99b36444bb811486be77100479 P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:43.468497 5440 raft_consensus.cc:3037] T a3445b99b36444bb811486be77100479 P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.473587 7441 raft_consensus.cc:2781] T 14d9bd97e4674e0ebdd90bf45b5679cb P 9808124b81e14d1290ab0a6641a4b836 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:13:43.473758 7441 raft_consensus.cc:686] T 14d9bd97e4674e0ebdd90bf45b5679cb P 9808124b81e14d1290ab0a6641a4b836 [term 3 LEADER]: Becoming Leader. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:43.474139 7441 consensus_queue.cc:237] T 14d9bd97e4674e0ebdd90bf45b5679cb P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.475832 5050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3445b99b36444bb811486be77100479" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:43.475908 5440 raft_consensus.cc:2396] T a3445b99b36444bb811486be77100479 P 2c4d718738d1408eabce489f4e140af6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9808124b81e14d1290ab0a6641a4b836 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:13:43.476069 5050 raft_consensus.cc:3037] T a3445b99b36444bb811486be77100479 P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.476645 5442 consensus_queue.cc:260] T 14d9bd97e4674e0ebdd90bf45b5679cb P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.477262 5442 raft_consensus.cc:3037] T 14d9bd97e4674e0ebdd90bf45b5679cb P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.479368 5050 raft_consensus.cc:2454] T a3445b99b36444bb811486be77100479 P b6cc443a494e4595b0e7cf1777620a3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 3.
I20240501 08:13:43.480666 5309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3fa99ffab5a44c588738be4476ed0fdc"
dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
from {username='slave'} at 127.0.0.1:57234
I20240501 08:13:43.481042 5309 raft_consensus.cc:465] T 3fa99ffab5a44c588738be4476ed0fdc P 9808124b81e14d1290ab0a6641a4b836 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 08:13:43.481220 5442 raft_consensus.cc:2396] T 14d9bd97e4674e0ebdd90bf45b5679cb P 2c4d718738d1408eabce489f4e140af6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9808124b81e14d1290ab0a6641a4b836 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:13:43.481204 7139 raft_consensus.cc:686] T ab68600377eb4581a724b714cfe234d7 P 9808124b81e14d1290ab0a6641a4b836 [term 2 LEADER]: Becoming Leader. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:43.481365 5247 leader_election.cc:302] T a3445b99b36444bb811486be77100479 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9808124b81e14d1290ab0a6641a4b836, b6cc443a494e4595b0e7cf1777620a3f; no voters: 2c4d718738d1408eabce489f4e140af6
I20240501 08:13:43.481743 7440 raft_consensus.cc:2781] T a3445b99b36444bb811486be77100479 P 9808124b81e14d1290ab0a6641a4b836 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:13:43.482555 7440 raft_consensus.cc:686] T a3445b99b36444bb811486be77100479 P 9808124b81e14d1290ab0a6641a4b836 [term 3 LEADER]: Becoming Leader. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:43.482618 5309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8a96986cf35c4720991739af0626e1e6"
dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
from {username='slave'} at 127.0.0.1:57234
I20240501 08:13:43.482930 5309 raft_consensus.cc:482] T 8a96986cf35c4720991739af0626e1e6 P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.483078 5309 raft_consensus.cc:3037] T 8a96986cf35c4720991739af0626e1e6 P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.481568 7139 consensus_queue.cc:237] T ab68600377eb4581a724b714cfe234d7 P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:43.487192 5309 raft_consensus.cc:504] T 8a96986cf35c4720991739af0626e1e6 P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:43.488178 5309 leader_election.cc:288] T 8a96986cf35c4720991739af0626e1e6 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 2 election: Requested vote from peers 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797), b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417)
I20240501 08:13:43.488502 7440 consensus_queue.cc:237] T a3445b99b36444bb811486be77100479 P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:43.488699 5050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a96986cf35c4720991739af0626e1e6" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:43.488910 5050 raft_consensus.cc:3037] T 8a96986cf35c4720991739af0626e1e6 P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.489115 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a96986cf35c4720991739af0626e1e6" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:43.489387 5442 raft_consensus.cc:3032] T 8a96986cf35c4720991739af0626e1e6 P 2c4d718738d1408eabce489f4e140af6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:43.489562 5442 raft_consensus.cc:729] T 8a96986cf35c4720991739af0626e1e6 P 2c4d718738d1408eabce489f4e140af6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:43.489908 5442 consensus_queue.cc:260] T 8a96986cf35c4720991739af0626e1e6 P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:43.491197 5442 raft_consensus.cc:3037] T 8a96986cf35c4720991739af0626e1e6 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.492069 5050 raft_consensus.cc:2454] T 8a96986cf35c4720991739af0626e1e6 P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 2.
I20240501 08:13:43.493237 5247 leader_election.cc:302] T 8a96986cf35c4720991739af0626e1e6 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9808124b81e14d1290ab0a6641a4b836, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:43.493868 7441 raft_consensus.cc:2781] T 8a96986cf35c4720991739af0626e1e6 P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:43.493964 5309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "51b04d461eed462fa3660c27d75d05b7"
dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
from {username='slave'} at 127.0.0.1:57234
I20240501 08:13:43.494127 7441 raft_consensus.cc:686] T 8a96986cf35c4720991739af0626e1e6 P 9808124b81e14d1290ab0a6641a4b836 [term 2 LEADER]: Becoming Leader. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:43.494185 5309 raft_consensus.cc:465] T 51b04d461eed462fa3660c27d75d05b7 P 9808124b81e14d1290ab0a6641a4b836 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 08:13:43.495368 5309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3d668173d46d48ff840a535273009626"
dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
from {username='slave'} at 127.0.0.1:57234
I20240501 08:13:43.495606 5309 raft_consensus.cc:482] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.495723 5309 raft_consensus.cc:3037] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.495633 4909 catalog_manager.cc:5484] T 19585175235c4877b9c4da931c345565 P 9808124b81e14d1290ab0a6641a4b836 reported cstate change: term changed from 2 to 3, leader changed from 2c4d718738d1408eabce489f4e140af6 (127.21.84.132) to 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131). New cstate: current_term: 3 leader_uuid: "9808124b81e14d1290ab0a6641a4b836" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:43.498427 5442 raft_consensus.cc:2454] T 8a96986cf35c4720991739af0626e1e6 P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 2.
I20240501 08:13:43.500753 7441 consensus_queue.cc:237] T 8a96986cf35c4720991739af0626e1e6 P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:43.500846 5309 raft_consensus.cc:504] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:43.501719 5309 leader_election.cc:288] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 3 election: Requested vote from peers 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797), b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417)
I20240501 08:13:43.502938 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d668173d46d48ff840a535273009626" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:43.503228 5051 raft_consensus.cc:3037] T 3d668173d46d48ff840a535273009626 P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.503787 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d668173d46d48ff840a535273009626" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:43.504072 5442 raft_consensus.cc:3032] T 3d668173d46d48ff840a535273009626 P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:13:43.504184 5442 raft_consensus.cc:729] T 3d668173d46d48ff840a535273009626 P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:43.504526 5442 consensus_queue.cc:260] T 3d668173d46d48ff840a535273009626 P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:43.504825 5442 raft_consensus.cc:3037] T 3d668173d46d48ff840a535273009626 P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.507283 5051 raft_consensus.cc:2454] T 3d668173d46d48ff840a535273009626 P b6cc443a494e4595b0e7cf1777620a3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 3.
I20240501 08:13:43.508069 5442 raft_consensus.cc:2396] T 3d668173d46d48ff840a535273009626 P 2c4d718738d1408eabce489f4e140af6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9808124b81e14d1290ab0a6641a4b836 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:13:43.508029 5247 leader_election.cc:302] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9808124b81e14d1290ab0a6641a4b836, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:43.508741 7440 raft_consensus.cc:2781] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:13:43.508926 7440 raft_consensus.cc:686] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836 [term 3 LEADER]: Becoming Leader. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:43.509199 7440 consensus_queue.cc:237] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:43.511086 5309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5169fa71adfc4a9d8c9feae475f2b14b"
dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
from {username='slave'} at 127.0.0.1:57234
I20240501 08:13:43.511293 5309 raft_consensus.cc:482] T 5169fa71adfc4a9d8c9feae475f2b14b P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.511430 5309 raft_consensus.cc:3037] T 5169fa71adfc4a9d8c9feae475f2b14b P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.512188 4909 catalog_manager.cc:5484] T ab68600377eb4581a724b714cfe234d7 P 9808124b81e14d1290ab0a6641a4b836 reported cstate change: term changed from 1 to 2, leader changed from 2c4d718738d1408eabce489f4e140af6 (127.21.84.132) to 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131). New cstate: current_term: 2 leader_uuid: "9808124b81e14d1290ab0a6641a4b836" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:43.512725 4909 catalog_manager.cc:5484] T faf43fec190c4830957d4e3ab304ba9b P 9808124b81e14d1290ab0a6641a4b836 reported cstate change: term changed from 1 to 2, leader changed from 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130) to 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131). New cstate: current_term: 2 leader_uuid: "9808124b81e14d1290ab0a6641a4b836" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: HEALTHY } } }
I20240501 08:13:43.513082 4909 catalog_manager.cc:5484] T a3445b99b36444bb811486be77100479 P 9808124b81e14d1290ab0a6641a4b836 reported cstate change: term changed from 2 to 3, leader changed from 2c4d718738d1408eabce489f4e140af6 (127.21.84.132) to 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131). New cstate: current_term: 3 leader_uuid: "9808124b81e14d1290ab0a6641a4b836" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: HEALTHY } } }
I20240501 08:13:43.513415 4909 catalog_manager.cc:5484] T c1bd10f523444d668736a2ff5e02442c P 9808124b81e14d1290ab0a6641a4b836 reported cstate change: term changed from 1 to 2, leader changed from 2c4d718738d1408eabce489f4e140af6 (127.21.84.132) to 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131). New cstate: current_term: 2 leader_uuid: "9808124b81e14d1290ab0a6641a4b836" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:43.513870 4909 catalog_manager.cc:5484] T 8a96986cf35c4720991739af0626e1e6 P 9808124b81e14d1290ab0a6641a4b836 reported cstate change: term changed from 1 to 2, leader changed from 2c4d718738d1408eabce489f4e140af6 (127.21.84.132) to 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131). New cstate: current_term: 2 leader_uuid: "9808124b81e14d1290ab0a6641a4b836" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:43.514317 4909 catalog_manager.cc:5484] T 14d9bd97e4674e0ebdd90bf45b5679cb P 9808124b81e14d1290ab0a6641a4b836 reported cstate change: term changed from 2 to 3, leader changed from 2c4d718738d1408eabce489f4e140af6 (127.21.84.132) to 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131). New cstate: current_term: 3 leader_uuid: "9808124b81e14d1290ab0a6641a4b836" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:43.514400 5309 raft_consensus.cc:504] T 5169fa71adfc4a9d8c9feae475f2b14b P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:43.515758 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5169fa71adfc4a9d8c9feae475f2b14b" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:43.515746 5181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5169fa71adfc4a9d8c9feae475f2b14b" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:43.515993 5181 raft_consensus.cc:3037] T 5169fa71adfc4a9d8c9feae475f2b14b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.516036 5051 raft_consensus.cc:3032] T 5169fa71adfc4a9d8c9feae475f2b14b P b6cc443a494e4595b0e7cf1777620a3f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:43.516227 5051 raft_consensus.cc:729] T 5169fa71adfc4a9d8c9feae475f2b14b P b6cc443a494e4595b0e7cf1777620a3f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b6cc443a494e4595b0e7cf1777620a3f, State: Running, Role: LEADER
I20240501 08:13:43.517853 5051 consensus_queue.cc:260] T 5169fa71adfc4a9d8c9feae475f2b14b P b6cc443a494e4595b0e7cf1777620a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:43.518306 5181 raft_consensus.cc:2454] T 5169fa71adfc4a9d8c9feae475f2b14b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 2.
I20240501 08:13:43.518391 5051 raft_consensus.cc:3037] T 5169fa71adfc4a9d8c9feae475f2b14b P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.518991 5246 leader_election.cc:302] T 5169fa71adfc4a9d8c9feae475f2b14b P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 9808124b81e14d1290ab0a6641a4b836; no voters:
I20240501 08:13:43.519340 7440 raft_consensus.cc:2781] T 5169fa71adfc4a9d8c9feae475f2b14b P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:43.519362 5309 leader_election.cc:288] T 5169fa71adfc4a9d8c9feae475f2b14b P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 2 election: Requested vote from peers b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417), 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937)
I20240501 08:13:43.519521 7440 raft_consensus.cc:686] T 5169fa71adfc4a9d8c9feae475f2b14b P 9808124b81e14d1290ab0a6641a4b836 [term 2 LEADER]: Becoming Leader. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:43.519794 7440 consensus_queue.cc:237] T 5169fa71adfc4a9d8c9feae475f2b14b P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:43.520375 5309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ff3b86014f2940b18707803c38bb5b8f"
dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
from {username='slave'} at 127.0.0.1:57234
I20240501 08:13:43.520606 5309 raft_consensus.cc:482] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.520716 5309 raft_consensus.cc:3037] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.523566 5309 raft_consensus.cc:504] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.524050 5051 raft_consensus.cc:2454] T 5169fa71adfc4a9d8c9feae475f2b14b P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 2.
I20240501 08:13:43.524703 5181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff3b86014f2940b18707803c38bb5b8f" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:43.524740 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff3b86014f2940b18707803c38bb5b8f" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:43.524925 5442 raft_consensus.cc:3032] T ff3b86014f2940b18707803c38bb5b8f P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:13:43.524926 5181 raft_consensus.cc:3037] T ff3b86014f2940b18707803c38bb5b8f P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.525034 5442 raft_consensus.cc:729] T ff3b86014f2940b18707803c38bb5b8f P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:43.525336 5442 consensus_queue.cc:260] T ff3b86014f2940b18707803c38bb5b8f P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.525635 5442 raft_consensus.cc:3037] T ff3b86014f2940b18707803c38bb5b8f P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.527726 5181 raft_consensus.cc:2454] T ff3b86014f2940b18707803c38bb5b8f P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 3.
I20240501 08:13:43.528237 5246 leader_election.cc:302] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 9808124b81e14d1290ab0a6641a4b836; no voters:
I20240501 08:13:43.528362 5442 raft_consensus.cc:2396] T ff3b86014f2940b18707803c38bb5b8f P 2c4d718738d1408eabce489f4e140af6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9808124b81e14d1290ab0a6641a4b836 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:13:43.528584 7440 raft_consensus.cc:2781] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:13:43.528772 7440 raft_consensus.cc:686] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836 [term 3 LEADER]: Becoming Leader. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:43.528816 5309 leader_election.cc:288] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 3 election: Requested vote from peers 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937), 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797)
I20240501 08:13:43.528561 4909 catalog_manager.cc:5484] T 3d668173d46d48ff840a535273009626 P 9808124b81e14d1290ab0a6641a4b836 reported cstate change: term changed from 2 to 3, leader changed from 2c4d718738d1408eabce489f4e140af6 (127.21.84.132) to 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131). New cstate: current_term: 3 leader_uuid: "9808124b81e14d1290ab0a6641a4b836" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:43.529047 7440 consensus_queue.cc:237] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.529219 4909 catalog_manager.cc:5484] T 5169fa71adfc4a9d8c9feae475f2b14b P 9808124b81e14d1290ab0a6641a4b836 reported cstate change: term changed from 1 to 2, leader changed from b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129) to 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131). New cstate: current_term: 2 leader_uuid: "9808124b81e14d1290ab0a6641a4b836" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:43.532117 5309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4c29ea3b807d4da1b970246c64797a91"
dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
from {username='slave'} at 127.0.0.1:57234
I20240501 08:13:43.532378 5309 raft_consensus.cc:482] T 4c29ea3b807d4da1b970246c64797a91 P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.532505 5309 raft_consensus.cc:3037] T 4c29ea3b807d4da1b970246c64797a91 P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.535329 5309 raft_consensus.cc:504] T 4c29ea3b807d4da1b970246c64797a91 P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:43.535470 4909 catalog_manager.cc:5484] T ff3b86014f2940b18707803c38bb5b8f P 9808124b81e14d1290ab0a6641a4b836 reported cstate change: term changed from 2 to 3, leader changed from 2c4d718738d1408eabce489f4e140af6 (127.21.84.132) to 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131). New cstate: current_term: 3 leader_uuid: "9808124b81e14d1290ab0a6641a4b836" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:43.536007 5309 leader_election.cc:288] T 4c29ea3b807d4da1b970246c64797a91 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 2 election: Requested vote from peers 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797), 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937)
I20240501 08:13:43.536571 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c29ea3b807d4da1b970246c64797a91" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:43.536785 5442 raft_consensus.cc:3032] T 4c29ea3b807d4da1b970246c64797a91 P 2c4d718738d1408eabce489f4e140af6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:43.536878 5442 raft_consensus.cc:729] T 4c29ea3b807d4da1b970246c64797a91 P 2c4d718738d1408eabce489f4e140af6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:43.536968 5181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c29ea3b807d4da1b970246c64797a91" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:43.537215 5181 raft_consensus.cc:3037] T 4c29ea3b807d4da1b970246c64797a91 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.537132 5442 consensus_queue.cc:260] T 4c29ea3b807d4da1b970246c64797a91 P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:43.537509 5442 raft_consensus.cc:3037] T 4c29ea3b807d4da1b970246c64797a91 P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.540095 5442 raft_consensus.cc:2454] T 4c29ea3b807d4da1b970246c64797a91 P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 2.
I20240501 08:13:43.540163 5181 raft_consensus.cc:2454] T 4c29ea3b807d4da1b970246c64797a91 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 2.
I20240501 08:13:43.540802 5248 leader_election.cc:302] T 4c29ea3b807d4da1b970246c64797a91 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4d718738d1408eabce489f4e140af6, 9808124b81e14d1290ab0a6641a4b836; no voters:
I20240501 08:13:43.542934 7440 raft_consensus.cc:2781] T 4c29ea3b807d4da1b970246c64797a91 P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:43.543220 7440 raft_consensus.cc:686] T 4c29ea3b807d4da1b970246c64797a91 P 9808124b81e14d1290ab0a6641a4b836 [term 2 LEADER]: Becoming Leader. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:43.543537 7440 consensus_queue.cc:237] T 4c29ea3b807d4da1b970246c64797a91 P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:43.544963 5309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c5de53e441514804b45ab7d1c30edfa5"
dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
from {username='slave'} at 127.0.0.1:57234
I20240501 08:13:43.545179 5309 raft_consensus.cc:482] T c5de53e441514804b45ab7d1c30edfa5 P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.545286 5309 raft_consensus.cc:3037] T c5de53e441514804b45ab7d1c30edfa5 P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.547780 4909 catalog_manager.cc:5484] T 4c29ea3b807d4da1b970246c64797a91 P 9808124b81e14d1290ab0a6641a4b836 reported cstate change: term changed from 1 to 2, leader changed from 2c4d718738d1408eabce489f4e140af6 (127.21.84.132) to 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131). New cstate: current_term: 2 leader_uuid: "9808124b81e14d1290ab0a6641a4b836" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: HEALTHY } } }
I20240501 08:13:43.548763 5309 raft_consensus.cc:504] T c5de53e441514804b45ab7d1c30edfa5 P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:43.556592 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5de53e441514804b45ab7d1c30edfa5" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:43.556592 5181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5de53e441514804b45ab7d1c30edfa5" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:43.556828 5181 raft_consensus.cc:3032] T c5de53e441514804b45ab7d1c30edfa5 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:43.556928 5051 raft_consensus.cc:3037] T c5de53e441514804b45ab7d1c30edfa5 P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.556939 5181 raft_consensus.cc:729] T c5de53e441514804b45ab7d1c30edfa5 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Running, Role: LEADER
I20240501 08:13:43.557222 5181 consensus_queue.cc:260] T c5de53e441514804b45ab7d1c30edfa5 P 0a952997a9fc4bfb8d7cbc56de6e0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:43.557621 5181 raft_consensus.cc:3037] T c5de53e441514804b45ab7d1c30edfa5 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.559984 5051 raft_consensus.cc:2454] T c5de53e441514804b45ab7d1c30edfa5 P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 2.
I20240501 08:13:43.560340 5181 raft_consensus.cc:2454] T c5de53e441514804b45ab7d1c30edfa5 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 2.
I20240501 08:13:43.560515 5247 leader_election.cc:302] T c5de53e441514804b45ab7d1c30edfa5 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9808124b81e14d1290ab0a6641a4b836, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:43.560880 7440 raft_consensus.cc:2781] T c5de53e441514804b45ab7d1c30edfa5 P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:43.560914 5309 leader_election.cc:288] T c5de53e441514804b45ab7d1c30edfa5 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 2 election: Requested vote from peers 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937), b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417)
I20240501 08:13:43.561079 7440 raft_consensus.cc:686] T c5de53e441514804b45ab7d1c30edfa5 P 9808124b81e14d1290ab0a6641a4b836 [term 2 LEADER]: Becoming Leader. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:43.561394 7440 consensus_queue.cc:237] T c5de53e441514804b45ab7d1c30edfa5 P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:43.562427 5309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0ad7f7e55fd34688b5eba16555306bc9"
dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
from {username='slave'} at 127.0.0.1:57234
I20240501 08:13:43.562672 5309 raft_consensus.cc:482] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.562825 5309 raft_consensus.cc:3037] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.565798 5309 raft_consensus.cc:504] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:43.567276 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ad7f7e55fd34688b5eba16555306bc9" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:43.567513 5051 raft_consensus.cc:3037] T 0ad7f7e55fd34688b5eba16555306bc9 P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.567579 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ad7f7e55fd34688b5eba16555306bc9" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:43.567799 5442 raft_consensus.cc:3032] T 0ad7f7e55fd34688b5eba16555306bc9 P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:13:43.567893 5442 raft_consensus.cc:729] T 0ad7f7e55fd34688b5eba16555306bc9 P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:43.568161 5442 consensus_queue.cc:260] T 0ad7f7e55fd34688b5eba16555306bc9 P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:43.568733 5442 raft_consensus.cc:3037] T 0ad7f7e55fd34688b5eba16555306bc9 P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.569543 7266 heartbeater.cc:503] Master 127.21.84.190:46613 was elected leader, sending a full tablet report...
I20240501 08:13:43.571265 5051 raft_consensus.cc:2454] T 0ad7f7e55fd34688b5eba16555306bc9 P b6cc443a494e4595b0e7cf1777620a3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 3.
I20240501 08:13:43.571676 5309 leader_election.cc:288] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 3 election: Requested vote from peers 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797), b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417)
I20240501 08:13:43.572878 5311 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2b611e855c3941769c4a26bcbe2b9b8c"
dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
from {username='slave'} at 127.0.0.1:57234
I20240501 08:13:43.573103 5311 raft_consensus.cc:482] T 2b611e855c3941769c4a26bcbe2b9b8c P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.573207 5311 raft_consensus.cc:3037] T 2b611e855c3941769c4a26bcbe2b9b8c P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.572878 5247 leader_election.cc:302] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9808124b81e14d1290ab0a6641a4b836, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:43.574646 7436 raft_consensus.cc:2781] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:13:43.574786 7436 raft_consensus.cc:686] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836 [term 3 LEADER]: Becoming Leader. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:43.575091 7436 consensus_queue.cc:237] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:43.575249 5442 raft_consensus.cc:2396] T 0ad7f7e55fd34688b5eba16555306bc9 P 2c4d718738d1408eabce489f4e140af6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9808124b81e14d1290ab0a6641a4b836 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:13:43.575909 5311 raft_consensus.cc:504] T 2b611e855c3941769c4a26bcbe2b9b8c P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:43.576944 5311 leader_election.cc:288] T 2b611e855c3941769c4a26bcbe2b9b8c P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 2 election: Requested vote from peers 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937), b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417)
I20240501 08:13:43.577179 5181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b611e855c3941769c4a26bcbe2b9b8c" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:43.577410 5181 raft_consensus.cc:3032] T 2b611e855c3941769c4a26bcbe2b9b8c P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:43.577505 5181 raft_consensus.cc:729] T 2b611e855c3941769c4a26bcbe2b9b8c P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Running, Role: LEADER
I20240501 08:13:43.577749 5181 consensus_queue.cc:260] T 2b611e855c3941769c4a26bcbe2b9b8c P 0a952997a9fc4bfb8d7cbc56de6e0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:43.578233 5181 raft_consensus.cc:3037] T 2b611e855c3941769c4a26bcbe2b9b8c P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.579013 5311 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e13f0b2e652c44fc994e9f950d40becb"
dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
from {username='slave'} at 127.0.0.1:57234
I20240501 08:13:43.579252 5311 raft_consensus.cc:482] T e13f0b2e652c44fc994e9f950d40becb P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.579418 5311 raft_consensus.cc:3037] T e13f0b2e652c44fc994e9f950d40becb P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.581573 5181 raft_consensus.cc:2454] T 2b611e855c3941769c4a26bcbe2b9b8c P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 2.
I20240501 08:13:43.582150 5246 leader_election.cc:302] T 2b611e855c3941769c4a26bcbe2b9b8c P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 9808124b81e14d1290ab0a6641a4b836; no voters:
I20240501 08:13:43.583338 7440 raft_consensus.cc:2781] T 2b611e855c3941769c4a26bcbe2b9b8c P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:43.583650 7440 raft_consensus.cc:686] T 2b611e855c3941769c4a26bcbe2b9b8c P 9808124b81e14d1290ab0a6641a4b836 [term 2 LEADER]: Becoming Leader. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:43.584048 7440 consensus_queue.cc:237] T 2b611e855c3941769c4a26bcbe2b9b8c P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } }
I20240501 08:13:43.584340 4909 catalog_manager.cc:5484] T c5de53e441514804b45ab7d1c30edfa5 P 9808124b81e14d1290ab0a6641a4b836 reported cstate change: term changed from 1 to 2, leader changed from 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130) to 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131). New cstate: current_term: 2 leader_uuid: "9808124b81e14d1290ab0a6641a4b836" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:43.585651 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b611e855c3941769c4a26bcbe2b9b8c" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:43.585896 5051 raft_consensus.cc:3037] T 2b611e855c3941769c4a26bcbe2b9b8c P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.591007 5051 raft_consensus.cc:2454] T 2b611e855c3941769c4a26bcbe2b9b8c P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 2.
I20240501 08:13:43.594203 5311 raft_consensus.cc:504] T e13f0b2e652c44fc994e9f950d40becb P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.595717 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13f0b2e652c44fc994e9f950d40becb" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:43.596015 5311 leader_election.cc:288] T e13f0b2e652c44fc994e9f950d40becb P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 2 election: Requested vote from peers b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417), 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797)
I20240501 08:13:43.596417 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13f0b2e652c44fc994e9f950d40becb" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:43.596704 5442 raft_consensus.cc:3032] T e13f0b2e652c44fc994e9f950d40becb P 2c4d718738d1408eabce489f4e140af6 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:43.596844 5442 raft_consensus.cc:729] T e13f0b2e652c44fc994e9f950d40becb P 2c4d718738d1408eabce489f4e140af6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:43.597227 5442 consensus_queue.cc:260] T e13f0b2e652c44fc994e9f950d40becb P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.597276 5311 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f518dc436d1d4e9dbe0ec9bbaf17e5fd"
dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
from {username='slave'} at 127.0.0.1:57234
I20240501 08:13:43.597548 5311 raft_consensus.cc:482] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.597723 5442 raft_consensus.cc:3037] T e13f0b2e652c44fc994e9f950d40becb P 2c4d718738d1408eabce489f4e140af6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.597831 5311 raft_consensus.cc:3037] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.600824 5442 raft_consensus.cc:2454] T e13f0b2e652c44fc994e9f950d40becb P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 2.
I20240501 08:13:43.601219 5051 raft_consensus.cc:3037] T e13f0b2e652c44fc994e9f950d40becb P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.601164 5311 raft_consensus.cc:504] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 9808124b81e14d1290ab0a6641a4b836 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:43.601433 5248 leader_election.cc:302] T e13f0b2e652c44fc994e9f950d40becb P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4d718738d1408eabce489f4e140af6, 9808124b81e14d1290ab0a6641a4b836; no voters:
I20240501 08:13:43.601796 7436 raft_consensus.cc:2781] T e13f0b2e652c44fc994e9f950d40becb P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:43.602111 7436 raft_consensus.cc:686] T e13f0b2e652c44fc994e9f950d40becb P 9808124b81e14d1290ab0a6641a4b836 [term 2 LEADER]: Becoming Leader. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:43.602372 5311 leader_election.cc:288] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 3 election: Requested vote from peers 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797), 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937)
I20240501 08:13:43.602473 7436 consensus_queue.cc:237] T e13f0b2e652c44fc994e9f950d40becb P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.602937 5181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f518dc436d1d4e9dbe0ec9bbaf17e5fd" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:43.603310 5181 raft_consensus.cc:3037] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.603325 5311 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f8f72bb32e34420eab300223e4428591"
dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
from {username='slave'} at 127.0.0.1:57234
I20240501 08:13:43.603538 5311 raft_consensus.cc:482] T f8f72bb32e34420eab300223e4428591 P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.603659 5311 raft_consensus.cc:3037] T f8f72bb32e34420eab300223e4428591 P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.606493 5181 raft_consensus.cc:2454] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 3.
I20240501 08:13:43.607038 5246 leader_election.cc:302] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 9808124b81e14d1290ab0a6641a4b836; no voters:
I20240501 08:13:43.607383 7441 raft_consensus.cc:2781] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 9808124b81e14d1290ab0a6641a4b836 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:13:43.607528 7441 raft_consensus.cc:686] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 9808124b81e14d1290ab0a6641a4b836 [term 3 LEADER]: Becoming Leader. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:43.607818 7441 consensus_queue.cc:237] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:43.608247 5311 raft_consensus.cc:504] T f8f72bb32e34420eab300223e4428591 P 9808124b81e14d1290ab0a6641a4b836 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:43.609261 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f518dc436d1d4e9dbe0ec9bbaf17e5fd" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:43.609573 5442 raft_consensus.cc:3032] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:13:43.609611 5440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8f72bb32e34420eab300223e4428591" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:43.609737 5442 raft_consensus.cc:729] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:43.609834 5440 raft_consensus.cc:3032] T f8f72bb32e34420eab300223e4428591 P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:13:43.609982 5440 raft_consensus.cc:729] T f8f72bb32e34420eab300223e4428591 P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:43.610339 5440 consensus_queue.cc:260] T f8f72bb32e34420eab300223e4428591 P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:43.610085 5442 consensus_queue.cc:260] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:43.610738 5440 raft_consensus.cc:3037] T f8f72bb32e34420eab300223e4428591 P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.610852 5442 raft_consensus.cc:3037] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.613117 5440 raft_consensus.cc:2396] T f8f72bb32e34420eab300223e4428591 P 2c4d718738d1408eabce489f4e140af6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9808124b81e14d1290ab0a6641a4b836 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:13:43.613734 5442 raft_consensus.cc:2396] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 2c4d718738d1408eabce489f4e140af6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9808124b81e14d1290ab0a6641a4b836 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:13:43.614925 5309 raft_consensus.cc:1261] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Refusing update from remote peer 2c4d718738d1408eabce489f4e140af6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:13:43.616248 7007 consensus_queue.cc:1035] T 19ec6a8b857d428d858fa831e825c578 P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:43.616587 5051 raft_consensus.cc:2454] T e13f0b2e652c44fc994e9f950d40becb P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 2.
I20240501 08:13:43.620954 4909 catalog_manager.cc:5484] T 0ad7f7e55fd34688b5eba16555306bc9 P 9808124b81e14d1290ab0a6641a4b836 reported cstate change: term changed from 2 to 3, leader changed from 2c4d718738d1408eabce489f4e140af6 (127.21.84.132) to 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131). New cstate: current_term: 3 leader_uuid: "9808124b81e14d1290ab0a6641a4b836" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:43.621800 4909 catalog_manager.cc:5484] T 2b611e855c3941769c4a26bcbe2b9b8c P 9808124b81e14d1290ab0a6641a4b836 reported cstate change: term changed from 1 to 2, leader changed from 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130) to 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131). New cstate: current_term: 2 leader_uuid: "9808124b81e14d1290ab0a6641a4b836" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:43.630064 5181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8f72bb32e34420eab300223e4428591" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:43.630935 5181 raft_consensus.cc:3037] T f8f72bb32e34420eab300223e4428591 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.634516 5180 raft_consensus.cc:1261] T 86ffaa5522df469e9bb410872fae256d P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Refusing update from remote peer 2c4d718738d1408eabce489f4e140af6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:13:43.635622 5311 leader_election.cc:288] T f8f72bb32e34420eab300223e4428591 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 3 election: Requested vote from peers 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797), 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937)
I20240501 08:13:43.635592 7007 consensus_queue.cc:1035] T 86ffaa5522df469e9bb410872fae256d P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:43.636261 5181 raft_consensus.cc:2454] T f8f72bb32e34420eab300223e4428591 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 3.
I20240501 08:13:43.636768 5246 leader_election.cc:302] T f8f72bb32e34420eab300223e4428591 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 9808124b81e14d1290ab0a6641a4b836; no voters: 2c4d718738d1408eabce489f4e140af6
I20240501 08:13:43.637785 5311 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "97943d9ec11f4c5da7a33a91818d9978"
dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
from {username='slave'} at 127.0.0.1:57234
I20240501 08:13:43.638079 5311 raft_consensus.cc:482] T 97943d9ec11f4c5da7a33a91818d9978 P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.638218 5311 raft_consensus.cc:3037] T 97943d9ec11f4c5da7a33a91818d9978 P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.638953 7436 raft_consensus.cc:2781] T f8f72bb32e34420eab300223e4428591 P 9808124b81e14d1290ab0a6641a4b836 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:13:43.639118 7436 raft_consensus.cc:686] T f8f72bb32e34420eab300223e4428591 P 9808124b81e14d1290ab0a6641a4b836 [term 3 LEADER]: Becoming Leader. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:43.639434 7436 consensus_queue.cc:237] T f8f72bb32e34420eab300223e4428591 P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:43.643093 5051 raft_consensus.cc:1261] T 86ffaa5522df469e9bb410872fae256d P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Refusing update from remote peer 2c4d718738d1408eabce489f4e140af6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:13:43.644606 7007 consensus_queue.cc:1035] T 86ffaa5522df469e9bb410872fae256d P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:43.648757 4909 catalog_manager.cc:5484] T e13f0b2e652c44fc994e9f950d40becb P 9808124b81e14d1290ab0a6641a4b836 reported cstate change: term changed from 1 to 2, leader changed from 2c4d718738d1408eabce489f4e140af6 (127.21.84.132) to 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131). New cstate: current_term: 2 leader_uuid: "9808124b81e14d1290ab0a6641a4b836" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:43.649770 4909 catalog_manager.cc:5484] T f518dc436d1d4e9dbe0ec9bbaf17e5fd P 9808124b81e14d1290ab0a6641a4b836 reported cstate change: term changed from 2 to 3, leader changed from 2c4d718738d1408eabce489f4e140af6 (127.21.84.132) to 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131). New cstate: current_term: 3 leader_uuid: "9808124b81e14d1290ab0a6641a4b836" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:43.650101 5311 raft_consensus.cc:504] T 97943d9ec11f4c5da7a33a91818d9978 P 9808124b81e14d1290ab0a6641a4b836 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.652578 5180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97943d9ec11f4c5da7a33a91818d9978" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:43.652894 5180 raft_consensus.cc:3037] T 97943d9ec11f4c5da7a33a91818d9978 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.656701 5180 raft_consensus.cc:2454] T 97943d9ec11f4c5da7a33a91818d9978 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 3.
I20240501 08:13:43.657354 5246 leader_election.cc:302] T 97943d9ec11f4c5da7a33a91818d9978 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 9808124b81e14d1290ab0a6641a4b836; no voters:
I20240501 08:13:43.657748 7441 raft_consensus.cc:2781] T 97943d9ec11f4c5da7a33a91818d9978 P 9808124b81e14d1290ab0a6641a4b836 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:13:43.657953 7441 raft_consensus.cc:686] T 97943d9ec11f4c5da7a33a91818d9978 P 9808124b81e14d1290ab0a6641a4b836 [term 3 LEADER]: Becoming Leader. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:43.658285 7441 consensus_queue.cc:237] T 97943d9ec11f4c5da7a33a91818d9978 P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.661497 5440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97943d9ec11f4c5da7a33a91818d9978" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:43.661845 5440 raft_consensus.cc:3032] T 97943d9ec11f4c5da7a33a91818d9978 P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:13:43.662014 5440 raft_consensus.cc:729] T 97943d9ec11f4c5da7a33a91818d9978 P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:43.662456 5440 consensus_queue.cc:260] T 97943d9ec11f4c5da7a33a91818d9978 P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.662827 5440 raft_consensus.cc:3037] T 97943d9ec11f4c5da7a33a91818d9978 P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.669162 5440 raft_consensus.cc:2396] T 97943d9ec11f4c5da7a33a91818d9978 P 2c4d718738d1408eabce489f4e140af6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9808124b81e14d1290ab0a6641a4b836 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:13:43.670953 5051 raft_consensus.cc:1261] T 19ec6a8b857d428d858fa831e825c578 P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Refusing update from remote peer 2c4d718738d1408eabce489f4e140af6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:13:43.671901 7404 consensus_queue.cc:1035] T 19ec6a8b857d428d858fa831e825c578 P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:43.672327 5311 leader_election.cc:288] T 97943d9ec11f4c5da7a33a91818d9978 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 3 election: Requested vote from peers 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937), 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797)
I20240501 08:13:43.675146 5311 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "19ec6a8b857d428d858fa831e825c578"
dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
from {username='slave'} at 127.0.0.1:57234
I20240501 08:13:43.675805 5311 raft_consensus.cc:482] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.676219 5311 raft_consensus.cc:3037] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.689628 5311 raft_consensus.cc:504] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.691319 5050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ec6a8b857d428d858fa831e825c578" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:43.691730 5050 raft_consensus.cc:3037] T 19ec6a8b857d428d858fa831e825c578 P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.692335 4909 catalog_manager.cc:5484] T f8f72bb32e34420eab300223e4428591 P 9808124b81e14d1290ab0a6641a4b836 reported cstate change: term changed from 2 to 3, leader changed from 2c4d718738d1408eabce489f4e140af6 (127.21.84.132) to 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131). New cstate: current_term: 3 leader_uuid: "9808124b81e14d1290ab0a6641a4b836" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:43.695780 5050 raft_consensus.cc:2454] T 19ec6a8b857d428d858fa831e825c578 P b6cc443a494e4595b0e7cf1777620a3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 3.
I20240501 08:13:43.696316 5247 leader_election.cc:302] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9808124b81e14d1290ab0a6641a4b836, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:43.696668 7436 raft_consensus.cc:2781] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:13:43.696900 7436 raft_consensus.cc:686] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836 [term 3 LEADER]: Becoming Leader. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:43.697244 7436 consensus_queue.cc:237] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.697571 5440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19ec6a8b857d428d858fa831e825c578" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:43.697877 5440 raft_consensus.cc:3032] T 19ec6a8b857d428d858fa831e825c578 P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:13:43.698191 5440 raft_consensus.cc:729] T 19ec6a8b857d428d858fa831e825c578 P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:43.698107 4909 catalog_manager.cc:5484] T 97943d9ec11f4c5da7a33a91818d9978 P 9808124b81e14d1290ab0a6641a4b836 reported cstate change: term changed from 2 to 3, leader changed from 2c4d718738d1408eabce489f4e140af6 (127.21.84.132) to 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131). New cstate: current_term: 3 leader_uuid: "9808124b81e14d1290ab0a6641a4b836" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:43.698581 5440 consensus_queue.cc:260] T 19ec6a8b857d428d858fa831e825c578 P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.699048 5440 raft_consensus.cc:3037] T 19ec6a8b857d428d858fa831e825c578 P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.699040 5051 raft_consensus.cc:1261] T e9e30e05c0ef498288f35706978c981b P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Refusing update from remote peer 2c4d718738d1408eabce489f4e140af6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:13:43.699692 7404 consensus_queue.cc:1035] T e9e30e05c0ef498288f35706978c981b P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:43.703212 5440 raft_consensus.cc:2454] T 19ec6a8b857d428d858fa831e825c578 P 2c4d718738d1408eabce489f4e140af6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 3.
I20240501 08:13:43.704016 5311 leader_election.cc:288] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 3 election: Requested vote from peers b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417), 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797)
I20240501 08:13:43.705454 5311 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e91cc538a8154588ac05a1e7e508b472"
dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
from {username='slave'} at 127.0.0.1:57234
I20240501 08:13:43.705235 4904 catalog_manager.cc:5484] T 19ec6a8b857d428d858fa831e825c578 P 9808124b81e14d1290ab0a6641a4b836 reported cstate change: term changed from 2 to 3, leader changed from 2c4d718738d1408eabce489f4e140af6 (127.21.84.132) to 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131). New cstate: current_term: 3 leader_uuid: "9808124b81e14d1290ab0a6641a4b836" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:43.705797 5311 raft_consensus.cc:482] T e91cc538a8154588ac05a1e7e508b472 P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.705968 5311 raft_consensus.cc:3037] T e91cc538a8154588ac05a1e7e508b472 P 9808124b81e14d1290ab0a6641a4b836 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.710026 5311 raft_consensus.cc:504] T e91cc538a8154588ac05a1e7e508b472 P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:43.711294 5180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e91cc538a8154588ac05a1e7e508b472" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
I20240501 08:13:43.711547 5180 raft_consensus.cc:3037] T e91cc538a8154588ac05a1e7e508b472 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.714576 5180 raft_consensus.cc:2454] T e91cc538a8154588ac05a1e7e508b472 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 2.
I20240501 08:13:43.715734 5246 leader_election.cc:302] T e91cc538a8154588ac05a1e7e508b472 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 9808124b81e14d1290ab0a6641a4b836; no voters:
I20240501 08:13:43.716162 7436 raft_consensus.cc:2781] T e91cc538a8154588ac05a1e7e508b472 P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:43.716312 7436 raft_consensus.cc:686] T e91cc538a8154588ac05a1e7e508b472 P 9808124b81e14d1290ab0a6641a4b836 [term 2 LEADER]: Becoming Leader. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:43.716630 7436 consensus_queue.cc:237] T e91cc538a8154588ac05a1e7e508b472 P 9808124b81e14d1290ab0a6641a4b836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:43.716470 5311 leader_election.cc:288] T e91cc538a8154588ac05a1e7e508b472 P 9808124b81e14d1290ab0a6641a4b836 [CANDIDATE]: Term 2 election: Requested vote from peers 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130:41937), b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417)
I20240501 08:13:43.717379 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e91cc538a8154588ac05a1e7e508b472" candidate_uuid: "9808124b81e14d1290ab0a6641a4b836" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:43.717605 5051 raft_consensus.cc:3032] T e91cc538a8154588ac05a1e7e508b472 P b6cc443a494e4595b0e7cf1777620a3f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:43.717686 5051 raft_consensus.cc:729] T e91cc538a8154588ac05a1e7e508b472 P b6cc443a494e4595b0e7cf1777620a3f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b6cc443a494e4595b0e7cf1777620a3f, State: Running, Role: LEADER
I20240501 08:13:43.717998 5051 consensus_queue.cc:260] T e91cc538a8154588ac05a1e7e508b472 P b6cc443a494e4595b0e7cf1777620a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:43.718377 5051 raft_consensus.cc:3037] T e91cc538a8154588ac05a1e7e508b472 P b6cc443a494e4595b0e7cf1777620a3f [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:43.721285 5051 raft_consensus.cc:2454] T e91cc538a8154588ac05a1e7e508b472 P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9808124b81e14d1290ab0a6641a4b836 in term 2.
I20240501 08:13:43.722882 5181 raft_consensus.cc:1261] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Refusing update from remote peer 2c4d718738d1408eabce489f4e140af6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:13:43.725248 5181 raft_consensus.cc:1261] T 98e5387cc69b483796e42cbf54f2226d P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Refusing update from remote peer 2c4d718738d1408eabce489f4e140af6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:13:43.725723 7405 consensus_queue.cc:1035] T e9e30e05c0ef498288f35706978c981b P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:43.730782 7405 consensus_queue.cc:1035] T 98e5387cc69b483796e42cbf54f2226d P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:43.731287 4904 catalog_manager.cc:5484] T e91cc538a8154588ac05a1e7e508b472 P 9808124b81e14d1290ab0a6641a4b836 reported cstate change: term changed from 1 to 2, leader changed from b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129) to 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131). New cstate: current_term: 2 leader_uuid: "9808124b81e14d1290ab0a6641a4b836" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: HEALTHY } } }
I20240501 08:13:43.737571 5051 raft_consensus.cc:1261] T 98e5387cc69b483796e42cbf54f2226d P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Refusing update from remote peer 2c4d718738d1408eabce489f4e140af6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:13:43.738198 7417 consensus_queue.cc:1035] T 98e5387cc69b483796e42cbf54f2226d P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:43.749861 5181 raft_consensus.cc:1261] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Refusing update from remote peer 2c4d718738d1408eabce489f4e140af6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:13:43.750577 7417 consensus_queue.cc:1035] T 2e0d1fd6767b454e9871e1bdec86caa8 P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:43.761472 5050 raft_consensus.cc:1261] T 2e0d1fd6767b454e9871e1bdec86caa8 P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Refusing update from remote peer 2c4d718738d1408eabce489f4e140af6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:13:43.762506 7404 consensus_queue.cc:1035] T 2e0d1fd6767b454e9871e1bdec86caa8 P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:43.771462 5181 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "19585175235c4877b9c4da931c345565"
dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
from {username='slave'} at 127.0.0.1:45028
I20240501 08:13:43.771759 5181 raft_consensus.cc:482] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.771844 5181 raft_consensus.cc:3037] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:13:43.774367 5181 raft_consensus.cc:504] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.775508 5181 leader_election.cc:288] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 4 election: Requested vote from peers 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983), 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797)
I20240501 08:13:43.775470 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19585175235c4877b9c4da931c345565" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
I20240501 08:13:43.775710 5311 raft_consensus.cc:3032] T 19585175235c4877b9c4da931c345565 P 9808124b81e14d1290ab0a6641a4b836 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:13:43.775830 5311 raft_consensus.cc:729] T 19585175235c4877b9c4da931c345565 P 9808124b81e14d1290ab0a6641a4b836 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:43.776043 5440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19585175235c4877b9c4da931c345565" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:43.776149 5311 consensus_queue.cc:260] T 19585175235c4877b9c4da931c345565 P 9808124b81e14d1290ab0a6641a4b836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.776290 5440 raft_consensus.cc:3037] T 19585175235c4877b9c4da931c345565 P 2c4d718738d1408eabce489f4e140af6 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:13:43.776474 5311 raft_consensus.cc:3037] T 19585175235c4877b9c4da931c345565 P 9808124b81e14d1290ab0a6641a4b836 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:13:43.778600 5440 raft_consensus.cc:2396] T 19585175235c4877b9c4da931c345565 P 2c4d718738d1408eabce489f4e140af6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a952997a9fc4bfb8d7cbc56de6e0598 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:13:43.779997 5311 raft_consensus.cc:2396] T 19585175235c4877b9c4da931c345565 P 9808124b81e14d1290ab0a6641a4b836 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a952997a9fc4bfb8d7cbc56de6e0598 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:13:43.779964 5181 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5e1018f0fee24311b58936ace2f6d53b"
dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
from {username='slave'} at 127.0.0.1:45028
I20240501 08:13:43.780160 5181 raft_consensus.cc:482] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.780330 5181 raft_consensus.cc:3037] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.780464 5117 leader_election.cc:302] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598; no voters: 2c4d718738d1408eabce489f4e140af6, 9808124b81e14d1290ab0a6641a4b836
I20240501 08:13:43.780823 6865 raft_consensus.cc:2726] T 19585175235c4877b9c4da931c345565 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 08:13:43.782642 5181 raft_consensus.cc:504] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:43.783370 5181 leader_election.cc:288] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 3 election: Requested vote from peers b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417), 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983)
I20240501 08:13:43.783762 5050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1018f0fee24311b58936ace2f6d53b" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:43.783880 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1018f0fee24311b58936ace2f6d53b" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
I20240501 08:13:43.784018 5050 raft_consensus.cc:3037] T 5e1018f0fee24311b58936ace2f6d53b P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.784035 5311 raft_consensus.cc:3032] T 5e1018f0fee24311b58936ace2f6d53b P 9808124b81e14d1290ab0a6641a4b836 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:13:43.784116 5311 raft_consensus.cc:729] T 5e1018f0fee24311b58936ace2f6d53b P 9808124b81e14d1290ab0a6641a4b836 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:43.784365 5311 consensus_queue.cc:260] T 5e1018f0fee24311b58936ace2f6d53b P 9808124b81e14d1290ab0a6641a4b836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:43.784580 5311 raft_consensus.cc:3037] T 5e1018f0fee24311b58936ace2f6d53b P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.786494 5050 raft_consensus.cc:2454] T 5e1018f0fee24311b58936ace2f6d53b P b6cc443a494e4595b0e7cf1777620a3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 3.
I20240501 08:13:43.786988 5117 leader_election.cc:302] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:43.787129 5181 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "faf43fec190c4830957d4e3ab304ba9b"
dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
from {username='slave'} at 127.0.0.1:45028
I20240501 08:13:43.787313 6865 raft_consensus.cc:2781] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:13:43.787365 5181 raft_consensus.cc:482] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.787479 5181 raft_consensus.cc:3037] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.787751 6865 raft_consensus.cc:686] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 3 LEADER]: Becoming Leader. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Running, Role: LEADER
I20240501 08:13:43.788148 6865 consensus_queue.cc:237] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:43.789335 5311 raft_consensus.cc:2396] T 5e1018f0fee24311b58936ace2f6d53b P 9808124b81e14d1290ab0a6641a4b836 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a952997a9fc4bfb8d7cbc56de6e0598 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:13:43.790481 5181 raft_consensus.cc:504] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:43.791302 5181 leader_election.cc:288] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 3 election: Requested vote from peers b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417), 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983)
I20240501 08:13:43.791802 5050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf43fec190c4830957d4e3ab304ba9b" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:43.792003 5050 raft_consensus.cc:3037] T faf43fec190c4830957d4e3ab304ba9b P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.792023 4904 catalog_manager.cc:5484] T 5e1018f0fee24311b58936ace2f6d53b P 0a952997a9fc4bfb8d7cbc56de6e0598 reported cstate change: term changed from 2 to 3, leader changed from 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131) to 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130). New cstate: current_term: 3 leader_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: HEALTHY } } }
I20240501 08:13:43.792428 5181 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5169fa71adfc4a9d8c9feae475f2b14b"
dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
from {username='slave'} at 127.0.0.1:45028
I20240501 08:13:43.792601 5181 raft_consensus.cc:482] T 5169fa71adfc4a9d8c9feae475f2b14b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.792711 5181 raft_consensus.cc:3037] T 5169fa71adfc4a9d8c9feae475f2b14b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.793673 5311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf43fec190c4830957d4e3ab304ba9b" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
I20240501 08:13:43.794661 5311 raft_consensus.cc:3032] T faf43fec190c4830957d4e3ab304ba9b P 9808124b81e14d1290ab0a6641a4b836 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:13:43.794787 5050 raft_consensus.cc:2454] T faf43fec190c4830957d4e3ab304ba9b P b6cc443a494e4595b0e7cf1777620a3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 3.
I20240501 08:13:43.794817 5311 raft_consensus.cc:729] T faf43fec190c4830957d4e3ab304ba9b P 9808124b81e14d1290ab0a6641a4b836 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:43.795218 5117 leader_election.cc:302] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:43.795509 6865 raft_consensus.cc:2781] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:13:43.795637 6865 raft_consensus.cc:686] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 3 LEADER]: Becoming Leader. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Running, Role: LEADER
I20240501 08:13:43.795892 6865 consensus_queue.cc:237] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:43.795948 5311 consensus_queue.cc:260] T faf43fec190c4830957d4e3ab304ba9b P 9808124b81e14d1290ab0a6641a4b836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } }
I20240501 08:13:43.796280 5311 raft_consensus.cc:3037] T faf43fec190c4830957d4e3ab304ba9b P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.799021 5311 raft_consensus.cc:2396] T faf43fec190c4830957d4e3ab304ba9b P 9808124b81e14d1290ab0a6641a4b836 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a952997a9fc4bfb8d7cbc56de6e0598 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:13:43.799018 5181 raft_consensus.cc:504] T 5169fa71adfc4a9d8c9feae475f2b14b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:43.799224 4904 catalog_manager.cc:5484] T faf43fec190c4830957d4e3ab304ba9b P 0a952997a9fc4bfb8d7cbc56de6e0598 reported cstate change: term changed from 2 to 3, leader changed from 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131) to 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130). New cstate: current_term: 3 leader_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: UNKNOWN } } }
I20240501 08:13:43.799871 5181 leader_election.cc:288] T 5169fa71adfc4a9d8c9feae475f2b14b P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 3 election: Requested vote from peers b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417), 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983)
I20240501 08:13:43.800107 5050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5169fa71adfc4a9d8c9feae475f2b14b" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:43.800350 5050 raft_consensus.cc:3037] T 5169fa71adfc4a9d8c9feae475f2b14b P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.800859 5181 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c1bd10f523444d668736a2ff5e02442c"
dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
from {username='slave'} at 127.0.0.1:45028
I20240501 08:13:43.801036 5181 raft_consensus.cc:482] T c1bd10f523444d668736a2ff5e02442c P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.801143 5181 raft_consensus.cc:3037] T c1bd10f523444d668736a2ff5e02442c P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.802382 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5169fa71adfc4a9d8c9feae475f2b14b" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
I20240501 08:13:43.802561 5309 raft_consensus.cc:3032] T 5169fa71adfc4a9d8c9feae475f2b14b P 9808124b81e14d1290ab0a6641a4b836 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:13:43.802641 5309 raft_consensus.cc:729] T 5169fa71adfc4a9d8c9feae475f2b14b P 9808124b81e14d1290ab0a6641a4b836 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:43.802881 5309 consensus_queue.cc:260] T 5169fa71adfc4a9d8c9feae475f2b14b P 9808124b81e14d1290ab0a6641a4b836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:43.803263 5309 raft_consensus.cc:3037] T 5169fa71adfc4a9d8c9feae475f2b14b P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.803248 5050 raft_consensus.cc:2454] T 5169fa71adfc4a9d8c9feae475f2b14b P b6cc443a494e4595b0e7cf1777620a3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 3.
I20240501 08:13:43.803279 5181 raft_consensus.cc:504] T c1bd10f523444d668736a2ff5e02442c P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.803946 5181 leader_election.cc:288] T c1bd10f523444d668736a2ff5e02442c P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 3 election: Requested vote from peers 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131:36983), 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797)
I20240501 08:13:43.804630 5440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1bd10f523444d668736a2ff5e02442c" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:43.804889 5440 raft_consensus.cc:3037] T c1bd10f523444d668736a2ff5e02442c P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.805667 5309 raft_consensus.cc:2396] T 5169fa71adfc4a9d8c9feae475f2b14b P 9808124b81e14d1290ab0a6641a4b836 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a952997a9fc4bfb8d7cbc56de6e0598 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:13:43.806402 5117 leader_election.cc:302] T 5169fa71adfc4a9d8c9feae475f2b14b P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, b6cc443a494e4595b0e7cf1777620a3f; no voters: 9808124b81e14d1290ab0a6641a4b836
I20240501 08:13:43.806707 6865 raft_consensus.cc:2781] T 5169fa71adfc4a9d8c9feae475f2b14b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:13:43.806867 6865 raft_consensus.cc:686] T 5169fa71adfc4a9d8c9feae475f2b14b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 3 LEADER]: Becoming Leader. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Running, Role: LEADER
I20240501 08:13:43.807202 5440 raft_consensus.cc:2454] T c1bd10f523444d668736a2ff5e02442c P 2c4d718738d1408eabce489f4e140af6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 3.
I20240501 08:13:43.807140 6865 consensus_queue.cc:237] T 5169fa71adfc4a9d8c9feae475f2b14b P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:43.807698 5118 leader_election.cc:302] T c1bd10f523444d668736a2ff5e02442c P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, 2c4d718738d1408eabce489f4e140af6; no voters:
I20240501 08:13:43.808003 7486 raft_consensus.cc:2781] T c1bd10f523444d668736a2ff5e02442c P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:13:43.808467 7486 raft_consensus.cc:686] T c1bd10f523444d668736a2ff5e02442c P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 3 LEADER]: Becoming Leader. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Running, Role: LEADER
I20240501 08:13:43.808931 5181 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2e0d1fd6767b454e9871e1bdec86caa8"
dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
from {username='slave'} at 127.0.0.1:45028
I20240501 08:13:43.808851 7486 consensus_queue.cc:237] T c1bd10f523444d668736a2ff5e02442c P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.809110 5181 raft_consensus.cc:482] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.809235 5181 raft_consensus.cc:3037] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.809962 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1bd10f523444d668736a2ff5e02442c" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9808124b81e14d1290ab0a6641a4b836"
I20240501 08:13:43.810146 5309 raft_consensus.cc:3032] T c1bd10f523444d668736a2ff5e02442c P 9808124b81e14d1290ab0a6641a4b836 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:13:43.810235 5309 raft_consensus.cc:729] T c1bd10f523444d668736a2ff5e02442c P 9808124b81e14d1290ab0a6641a4b836 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9808124b81e14d1290ab0a6641a4b836, State: Running, Role: LEADER
I20240501 08:13:43.810518 5309 consensus_queue.cc:260] T c1bd10f523444d668736a2ff5e02442c P 9808124b81e14d1290ab0a6641a4b836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.810823 5309 raft_consensus.cc:3037] T c1bd10f523444d668736a2ff5e02442c P 9808124b81e14d1290ab0a6641a4b836 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.812213 5181 raft_consensus.cc:504] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.813274 5181 leader_election.cc:288] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 3 election: Requested vote from peers b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417), 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797)
I20240501 08:13:43.813385 5050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0d1fd6767b454e9871e1bdec86caa8" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:43.813390 5440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0d1fd6767b454e9871e1bdec86caa8" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:43.813570 5050 raft_consensus.cc:3037] T 2e0d1fd6767b454e9871e1bdec86caa8 P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.813606 5440 raft_consensus.cc:3032] T 2e0d1fd6767b454e9871e1bdec86caa8 P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:13:43.813885 5181 raft_consensus.cc:1261] T 1835b9a1f198420c82e1994a3056ea9a P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Refusing update from remote peer 2c4d718738d1408eabce489f4e140af6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:13:43.813951 5440 raft_consensus.cc:729] T 2e0d1fd6767b454e9871e1bdec86caa8 P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:43.814546 7404 consensus_queue.cc:1035] T 1835b9a1f198420c82e1994a3056ea9a P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:43.814604 5181 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e9e30e05c0ef498288f35706978c981b"
dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
from {username='slave'} at 127.0.0.1:45028
I20240501 08:13:43.814786 5181 raft_consensus.cc:482] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.814908 5181 raft_consensus.cc:3037] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.815053 5440 consensus_queue.cc:260] T 2e0d1fd6767b454e9871e1bdec86caa8 P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.815536 5440 raft_consensus.cc:3037] T 2e0d1fd6767b454e9871e1bdec86caa8 P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.816470 5050 raft_consensus.cc:2454] T 2e0d1fd6767b454e9871e1bdec86caa8 P b6cc443a494e4595b0e7cf1777620a3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 3.
I20240501 08:13:43.817291 5117 leader_election.cc:302] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:43.817673 7490 raft_consensus.cc:2781] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:13:43.819517 5440 raft_consensus.cc:2454] T 2e0d1fd6767b454e9871e1bdec86caa8 P 2c4d718738d1408eabce489f4e140af6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 3.
I20240501 08:13:43.820850 5309 raft_consensus.cc:2396] T c1bd10f523444d668736a2ff5e02442c P 9808124b81e14d1290ab0a6641a4b836 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a952997a9fc4bfb8d7cbc56de6e0598 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:13:43.821789 7490 raft_consensus.cc:686] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 3 LEADER]: Becoming Leader. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Running, Role: LEADER
I20240501 08:13:43.822245 7490 consensus_queue.cc:237] T 2e0d1fd6767b454e9871e1bdec86caa8 P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.826361 5050 raft_consensus.cc:1261] T 1835b9a1f198420c82e1994a3056ea9a P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Refusing update from remote peer 2c4d718738d1408eabce489f4e140af6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:13:43.826932 5181 raft_consensus.cc:504] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.826951 7417 consensus_queue.cc:1035] T 1835b9a1f198420c82e1994a3056ea9a P 2c4d718738d1408eabce489f4e140af6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:43.828279 5049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e30e05c0ef498288f35706978c981b" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:43.828342 5440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e30e05c0ef498288f35706978c981b" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:43.828518 5049 raft_consensus.cc:3037] T e9e30e05c0ef498288f35706978c981b P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.828608 5440 raft_consensus.cc:3032] T e9e30e05c0ef498288f35706978c981b P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:13:43.828688 5440 raft_consensus.cc:729] T e9e30e05c0ef498288f35706978c981b P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:43.828954 5440 consensus_queue.cc:260] T e9e30e05c0ef498288f35706978c981b P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.829355 5440 raft_consensus.cc:3037] T e9e30e05c0ef498288f35706978c981b P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.831573 5049 raft_consensus.cc:2454] T e9e30e05c0ef498288f35706978c981b P b6cc443a494e4595b0e7cf1777620a3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 3.
I20240501 08:13:43.832087 5440 raft_consensus.cc:2454] T e9e30e05c0ef498288f35706978c981b P 2c4d718738d1408eabce489f4e140af6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 3.
I20240501 08:13:43.833045 5117 leader_election.cc:302] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a952997a9fc4bfb8d7cbc56de6e0598, b6cc443a494e4595b0e7cf1777620a3f; no voters:
I20240501 08:13:43.834214 5181 leader_election.cc:288] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 3 election: Requested vote from peers b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417), 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797)
I20240501 08:13:43.834753 6865 raft_consensus.cc:2781] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:13:43.834918 6865 raft_consensus.cc:686] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 3 LEADER]: Becoming Leader. State: Replica: 0a952997a9fc4bfb8d7cbc56de6e0598, State: Running, Role: LEADER
I20240501 08:13:43.835171 6865 consensus_queue.cc:237] T e9e30e05c0ef498288f35706978c981b P 0a952997a9fc4bfb8d7cbc56de6e0598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } } peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } }
I20240501 08:13:43.836546 5181 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "86ffaa5522df469e9bb410872fae256d"
dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
from {username='slave'} at 127.0.0.1:45028
I20240501 08:13:43.836726 5181 raft_consensus.cc:482] T 86ffaa5522df469e9bb410872fae256d P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:43.836839 5181 raft_consensus.cc:3037] T 86ffaa5522df469e9bb410872fae256d P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.836292 4904 catalog_manager.cc:5484] T 5169fa71adfc4a9d8c9feae475f2b14b P 0a952997a9fc4bfb8d7cbc56de6e0598 reported cstate change: term changed from 2 to 3, leader changed from 9808124b81e14d1290ab0a6641a4b836 (127.21.84.131) to 0a952997a9fc4bfb8d7cbc56de6e0598 (127.21.84.130). New cstate: current_term: 3 leader_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9808124b81e14d1290ab0a6641a4b836" member_type: VOTER last_known_addr { host: "127.21.84.131" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } health_report { overall_health: HEALTHY } } }
I20240501 08:13:43.839089 5181 raft_consensus.cc:504] T 86ffaa5522df469e9bb410872fae256d P 0a952997a9fc4bfb8d7cbc56de6e0598 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:43.839732 5181 leader_election.cc:288] T 86ffaa5522df469e9bb410872fae256d P 0a952997a9fc4bfb8d7cbc56de6e0598 [CANDIDATE]: Term 3 election: Requested vote from peers b6cc443a494e4595b0e7cf1777620a3f (127.21.84.129:35417), 2c4d718738d1408eabce489f4e140af6 (127.21.84.132:34797)
I20240501 08:13:43.840333 5051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ffaa5522df469e9bb410872fae256d" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b6cc443a494e4595b0e7cf1777620a3f"
I20240501 08:13:43.840540 5051 raft_consensus.cc:3037] T 86ffaa5522df469e9bb410872fae256d P b6cc443a494e4595b0e7cf1777620a3f [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.840795 5440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ffaa5522df469e9bb410872fae256d" candidate_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "2c4d718738d1408eabce489f4e140af6"
I20240501 08:13:43.841224 5440 raft_consensus.cc:3032] T 86ffaa5522df469e9bb410872fae256d P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:13:43.841377 5440 raft_consensus.cc:729] T 86ffaa5522df469e9bb410872fae256d P 2c4d718738d1408eabce489f4e140af6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2c4d718738d1408eabce489f4e140af6, State: Running, Role: LEADER
I20240501 08:13:43.841703 5440 consensus_queue.cc:260] T 86ffaa5522df469e9bb410872fae256d P 2c4d718738d1408eabce489f4e140af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6cc443a494e4595b0e7cf1777620a3f" member_type: VOTER last_known_addr { host: "127.21.84.129" port: 35417 } } peers { permanent_uuid: "2c4d718738d1408eabce489f4e140af6" member_type: VOTER last_known_addr { host: "127.21.84.132" port: 34797 } } peers { permanent_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598" member_type: VOTER last_known_addr { host: "127.21.84.130" port: 41937 } }
I20240501 08:13:43.842054 5440 raft_consensus.cc:3037] T 86ffaa5522df469e9bb410872fae256d P 2c4d718738d1408eabce489f4e140af6 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:13:43.843361 5051 raft_consensus.cc:2454] T 86ffaa5522df469e9bb410872fae256d P b6cc443a494e4595b0e7cf1777620a3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a952997a9fc4bfb8d7cbc56de6e0598 in term 3.
I20240501 08:13:43.843647 5181 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3fa99ffab5a44c588738be4476ed0fdc"
dest_uuid: "0a952997a9fc4bfb8d7cbc56de6e0598"
from {username='slave'